2D Image Analyzer

ABSTRACT

A 2D image analyzer includes an image scaler, an image generator and a pattern finder. The image scaler is configured to scale an image according to a scaling factor. The image generator is configured to produce an overview image including a plurality of copies of the received and scaled image, wherein every copy is scaled about a different scaling factor. Thereby, the respective position can be calculable by an algorithm, which considers a gap between the scaled images in the overview image, a gap of the scaled image towards one or more borders of the overview image and/or other predefined conditions. The pattern finder is configured to perform a feature transformation and classification of the overview image in order to output a position at which an accordance of the searched pattern and the predetermined pattern is maximal. A post-processing unit for smoothening and correcting the position of local maxima in the classified overview image may also be provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending InternationalApplication No. PCT/EP2015/052009, filed Jan. 30, 2015, which isincorporated herein by reference in its entirety, and additionallyclaims priority from German Application No. DE 102014201997.4, filedFeb. 4, 2014, which is incorporated herein by reference in its entirety.

Embodiments of the present invention relate to a 2D image analyzer aswell as to a corresponding method.

BACKGROUND OF THE INVENTION

In many cases of digital image processing, several scaled versions ofthe initial image are used for the pattern recognition. An example forsuch a pattern recognizer is the classifier according to Viola-Jones,which is trained for a specific model size.

Depending on which distance the searched pattern (e.g. face or eye) hastowards the camera system, a greater or smaller display occurs. Due tothe fixed model size of the classifier, thus, it has to be searched inseveral scaling stages of the recorded image in order to obtain in oneof the scaling stages an optimal accordance with the model size of theclassifier. The scaling stages are normally searched ascendingly ordescendingly (cf. image pyramid). This sequential processing is inparticular very poorly suited for parallel architectures (e.g. FPGA).

Therefore, there is the need for an improved concept.

SUMMARY

According to an embodiment, a 2D image analyzer may have: an imagescaler, which is configured to receive an image, which includes asearched pattern and to scale the received image according to a scalingfactor; an image generator, which is configured to produce an overviewimage, which includes a plurality of copies of the received and scaledimage, wherein each copy is scaled about a different scaling factor;wherein said producing includes compiling the copies of the received andscaled image to form the overview image and arranging the copies of thereceived and scaled image within the image matrix of the overview image,said image generator calculating, for the copies of the received andscaled image, a respective position within the overview image whiletaking into account a gap between the copies of the received and scaledimage within the overview image and a gap of the copies of the receivedand scaled image to one or more of the borders of the overview image;and a pattern finder, which is implemented on an FPGA with parallelarchitectures and is configured to compare a predetermined pattern withthe overview image and to output an information regarding the positionwithin the overview image, at which an accordance between the searchedpattern and the predetermined pattern is maximal, wherein the positionrelates to a respective copy of the received and scaled image, thepattern finder searching all of the scaling stages which are of interestall at once in one step only.

According to another embodiment, a 2D image analyzing system may havethe inventive 2D image analyzer, wherein the 2D image analyzer isconnected to an image analyzing system for collecting and/or tracking ofa pupil including a first Hough path for a first camera and a processingunit for post-processing of the first Hough path results, wherein thefirst Hough path includes a Hough processor which may have: apre-processor, which is configured to receive a plurality of samples,each including an image and in order to rotate and/or reflect the imageof the respective sample and in order to output a plurality of versionsof the image of the respective sample for each sample; and a Houghtransformation unit, which is configured to collect a predeterminedsearched pattern in the plurality of samples on the basis of theplurality of versions, wherein a characteristic being dependent on thesearched pattern of the Hough transformation unit is adjustable, whereinthe processing unit has a unit for analyzing the collected pattern andfor outputting a set of geometry parameters, which describes a positionand/or a geometry of the pattern for each sample.

According to another embodiment, a 2D image analyzing system may havethe inventive 2D image analyzer and an evaluation unit, wherein theevaluation unit is configured to detect an absent reaction of the eye.

According to another embodiment, a method for analyzing a 2D image mayhave the steps of: scaling a received image including a searched patternaccording to a scaling factor; producing an overview image including aplurality of copies of the received and scaled image, wherein every copyis scaled about a different scaling factor; wherein said producingincludes compiling the copies of the received and scaled image to formthe overview image and arranging the copies of the received and scaledimage within the image matrix of the overview image, said imagegenerator calculating, for the copies of the received and scaled image,a respective position within the overview image while taking intoaccount a gap between the copies of the received and scaled image withinthe overview image and a gap of the copies of the received and scaledimage to one or more of the borders of the overview image; andcomparing, by means of a pattern finder implemented on an FPGA withparallel architectures, a predetermined pattern with the overview imageand outputting an information in respect to a position within theoverview image at which an accordance between the searched pattern andthe predetermined pattern is maximal, wherein the position relates to arespective copy of the received and scaled image; the pattern findersearching all of the scaling stages which are of interest all at once inone step only.

According to another embodiment, a non-transitory digital storage mediummay have a computer program stored thereon to perform the method foranalyzing a 2D image which may have the steps of: scaling a receivedimage including a searched pattern according to a scaling factor;producing an overview image including a plurality of copies of thereceived and scaled image, wherein every copy is scaled about adifferent scaling factor; wherein said producing includes compiling thecopies of the received and scaled image to form the overview image andarranging the copies of the received and scaled image within the imagematrix of the overview image, said image generator calculating, for thecopies of the received and scaled image, a respective position withinthe overview image while taking into account a gap between the copies ofthe received and scaled image within the overview image and a gap of thecopies of the received and scaled image to one or more of the borders ofthe overview image; and comparing, by means of a pattern finderimplemented on an FPGA with parallel architectures, a predeterminedpattern with the overview image and outputting an information in respectto a position within the overview image at which an accordance betweenthe searched pattern and the predetermined pattern is maximal, whereinthe position relates to a respective copy of the received and scaledimage; the pattern finder searching all of the scaling stages which areof interest all at once in one step only, when said computer program isrun by a computer.

Embodiments create a 2D image analyzer with an image scaler, an imagegenerator and a pattern finder. The image scaler is configured in orderto receive an image having a searched pattern and in order to scale thereceived image according to a scaling factor. The image generator isconfigured in order to produce an overview image comprising a pluralityof copies of the received and scaled image, whereby every copy is scaledabout a different scaling factor. The pattern finder is configured inorder to compare a predetermined pattern with a plurality of receivedand scaled images within the overview image and in order to output aninformation in respect to a position, for which an accordance betweenthe searched pattern and the predetermined pattern is maximal, wherebythe position relates to the respective copy of the received and scaledimage.

The gist of the present invention, thus is the fact that therecognizability of specific patterns changes over the image size. Here,it was in particular realized that the sequential processing ofdifferent scaling stages can be very inefficient. Therefore, embodimentsof the present invention create an image analyzer with a pattern finder,whereby the pattern finder is applied to an overview image, whichincludes the image in which the searched pattern is contained, indifferent scaling stages. Due to these different scaling stages, whichare arranged in the overview image in a sent way, it can be achievedthat in only one step several or all interesting scaling stages can besearched at once (and not that as usual the pattern recognition isapplied on several scaling stages one after the other). This occurs bythe way that the pattern recognition only takes place on the overviewimage. At the point, where the overview image has the greatestaccordance with the searched pattern, it is obvious on the one handwhich scaling stage delivered this accordance (position of the scaledimage within the overview image) and on the other hand, at whichposition (x-, y-coordinate in the image) this lied (position within thescaled image in the overview image, corrected about the scaling factor).Hence, this in short offers the advantage that the searched pattern canbe collected considerably more efficient, in particular on parallelarchitectures, as e.g. FPGAs, by only performing the pattern recognitionon the overview image. Due to the knowledge of the scaling, also in theabsolute image, the position of the searched pattern is calculable.

Therefore, according to further embodiments, every scaled image isassigned according to the respective scaling factor to a respectiveposition in the overview image. Thereby, the respective position can becalculable by an algorithm which considers a gap between the scaledimages in the overview image, a gap of the scaled images towards one ormore borders in the overview image and/or other predefined conditions.

According to further embodiments, to this prepared overview image, thepattern finder is applied, which is configured in order to identify oneor more local maxima in the census transformed version of the overviewimage or in the version of the overview image transferred into the Houghfeature room or in the version of the overview image transferred in thegradient image, or generally, in the version of the overview imagetransferred in a feature room, whereby a position of a local maximumindicates the position of the identified predetermined pattern in therespective copy of the received and scaled image.

According to further embodiments, the pattern finder comprises aclassification and a post-processing. The classification is therebyapplied to the overview image transferred into the feature room andprovides high values (local maxima) at positions, where the imagecontent is in accordance with the searched pattern. For theclassification, therefor, the common methods according to the state ofthe art can be consulted (e.g. according to Viola-Jones).

The classified overview image now can correspondingly be subject to apost-processing. For this, the classified overview image is initiallysmoothened with a local amount filter and this way, also the positionsof the local maxima are locally corrected. Parallel to this, a localmaximum filter is used in order to obtain the score (a score in mostcases is the result of a classification, thus, a measure for theaccordance of the image content with the searched pattern), via theamount filter corrected, local maximum. As a result, again a classifiedoverview image with scores of the classifier, but now locally correctedlocal maxima. To every position of local maximum in the overview image,corresponding to the previous embodiments, a corresponding scaling stageis assigned (except at those positions, where the scaling stages arespaced one below the other and towards the borders of the overviewimage). Depending on where exactly within a scaling stage the maximumlies, this position can be retransferred to the original image, bycorrecting the corresponding position about the scaling factor of thecorresponding scaling stage. To every position in the overview image,thus, an absolute position in the original coordinates is assigned(except at those positions, where the scaling stages one below the otherand towards the borders of the overview image are spaced). If a maximumhad been extracted within a scaling stage, this position can also becorrected once again by consulting the corresponding local maximum fromthe adjacent scaling stages and an averaging via the adjacent scalingstages occurs.

Further embodiments relate to the exact implementation of the 2D imageanalyzer, which according to a first group of embodiments can bedesigned as an embedded processor or programmable logic or asclient-specific unit (ASIC), or according to a second group ofembodiments realized as a method, which runs on a computer (computerprogram).

Thus, one embodiment comprises the method for analyzing a 2D image withsteps: scaling of a received image, which comprises a searched patternaccording to a scaling factor and producing of an overview image, whichcomprises a plurality of copies of the received and scaled image,whereby every grouping is scaled about a different scaling factor. Asthe next step, a transfer of the overview image into a feature room(e.g. Hough feature room) takes place and then a classification in orderto determine a measure for the accordance with a predetermined pattern(as e.g. an eye). Then, maxima is searched in the plurality of receivedand scaled images within the overview image, in order to output aninformation in respect to a position, at which an accordance betweensearched pattern and predetermined pattern is maximal, whereby theposition relates to a respective copy of the received and scaled image.The position can, if need be, be corrected by a combination from localamount and maximum filters and averaged via adjacent scaling stages.

According to further embodiments, the 2D image analyzer can combinedwith a 2D image analyzing system, whereby the analyzing system then hasan evaluation unit, which monitors the predetermined pattern and inparticular the pupil (or, generally, the eye region or the eye of theuser) and determines a status to this. Thereby, it is in particularpossible to detect an absent reaction of the eye, as e.g. an eye, whichcannot be opened anymore, e.g. due to a momentary nodding off.

According to further embodiments, the 2D image analyzer can be connectedto a further processing unit, which comprises a selective adaptive dataprocessor, which is configured in order to exchange several sets of datawith the image analyzer and to process these data sets. Thereby, thedata sets are processed insofar that only plausible data sets aretransmitted, whereas implausible parts of the data sets are replaced byplausible parts.

According to further embodiments, the 2D image analyzer can also beconnected to a 3D image analyzer, which determines an alignment of anobject in the room (thus, e.g. a point of view) based on at least oneset of image data in combination with additional information. This 3Dimage analyzer comprises two main units, namely, a position calculatorfor determination of the position of the pattern in thethree-dimensional room and an alignment calculator for determination ofan axis passing exactly this pattern.

According to further embodiments, the 2D image analyzer can also beconnected with a Hough processor, which in turn is divided into thefollowing two sub-units: pre-processor, configured in order to receive aplurality of samples respectively comprising an image and in order torotate and/or reflect the image of the respective sample and in order tooutput a plurality of versions of the image for each sample of therespective sample. Hough transformation unit, configured in order tocollect a predetermined searched pattern in a plurality of samples onthe basis of the plurality of versions, whereby a characteristic beingdependent on the searched pattern is adjustable in the Houghtransformation unit. The processing unit for the post-processing of theHough results is configured in order to analyze the collected patternand to output a set of geometry parameters, which describes a positionand/or a geometry of the pattern for each sample.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 shows a schematic block diagram of a 2D image analyzer accordingto an embodiment;

FIG. 2a shows a schematic block diagram of a Hough processor with apre-processor and a Hough transformation unit according to anembodiment;

FIG. 2b shows a schematic block diagram of a pre-processor according toan embodiment;

FIG. 2c shows a schematic illustration of Hough cores for the detectionof straights (sections);

FIG. 3a shows a schematic block diagram of a possible implementation ofa Hough transformation unit according to an embodiment;

FIG. 3b shows a single cell of a deceleration matrix according to anembodiment;

FIG. 4a-d show a schematic block diagram of a further implementation ofa Hough transformation unit according to an embodiment;

FIG. 5a shows a schematic block diagram of a stereoscopic cameraassembly with two image processors and a post-processing unit, wherebyeach of the image processors comprises one Hough processor according toembodiments;

FIG. 5b shows an exemplary picture of an eye for the illustration of apoint of view detection, which is feasible with the unit from FIG. 5aand for explanation of the point of view detection in the monoscopiccase;

FIG. 6-7 a-c show further illustrations for explanation of additionalembodiments and/or aspects;

FIG. 8a-e show schematic illustrations of optical systems;

FIG. 9a-9i show further illustrations for explanation of backgroundknowledge for the Hough transformation unit.

DETAILED DESCRIPTION OF THE INVENTION

In the following, embodiments of the present invention are described indetail by means of the Figures. It should be noted that same elementsare provided with the same reference signs so that the description ofwhose is applicable to one another and/or is exchangeable.

FIG. 1 shows a 2D image analyzer 700 with an image scaler 702, an imagegenerator 702, and a pattern finder 706.

The image scaler 702 is configured in order to receive an image 710 (cf.FIG. 7b ), which comprises a searched pattern 711. In the image scaler702, the received image 710 is scaled with different scaling factors.Thereby, for example, a downsizing can occur, as shown in the following.During this downsizing, on the basis of the image 710, a plurality ofcopes of the received and scaled image 710′, 710″, 710′″ to 710′″″″″arise, whereby it should be noted that the number of scaling stages isnot limited. This plurality of scaled images 710′ to 710′″″″″ areassembled by means of the image generator 704 to an overview image 712,whereby the overview image 712 comprises the plurality of copies of thereceived and scaled image 710′ to 710′″″″″. Thereby, the respectiveposition of a scaled image within the overview image can be calculableby an algorithm, which considers a gap between the scaled images in theoverview image, a gap of the scaled image towards one or more borders ofthe overview image and/or other predefined conditions. In this overviewimage 712, by means of the pattern finder 706, the searched pattern 711now can be detected.

Thus, instead of applying the pattern recognition on several scalingstages one after the other, the pattern recognition is applied only onceto the overview image, whereby the respective scaling stage is “encoded”as location information within the overview image. This kind ofprocessing of individual scaling stages is in particular advantageous onFPGA architectures, as here sequential processing of individual scalingstages one after the other would be relatively complex. The differentscaled images would have to be respectively stored in the storage andseparately processed and then the results brought together. Thus, theoverview image is generated once and then can be processed in one step.This way, the parallel FPGA architectures can be exploited optimally.

According to further embodiments, it would also be possible to transformthis overview image 710 into a feature image 710 a, whereby typicallyhere a census transformation is used. Based on this census-transformedimage 710 a, it would also be conceivable to make a classification inorder to detect the false color image 710 b (cf. FIG. 7c ).

Optionally, according to further embodiments, additionally, apost-processing unit (not shown) can be provided in the 2D imageanalyzer (cf. FIG. 5a ). The post-processing unit allows an applicationof a combination from local amount and maximum filter, in order tocorrect the position of the local maxima in the classified overviewimage. Optionally, thereby, also an averaging of the position of amaximum, which was found in one of the scaling stages with thecorresponding maximum from the adjacent scaling stages can occur. Thus,an averaging over and away of adjacent scaling stages in the overviewimage occurs.

Further embodiments or variants of this 2D image analyzer are describedin respect to FIG. 7 a.

FIG. 2a shows a Hough processor 100 with a pre-processor 102 and a Houghtransformation unit 104. The pre-processor 102 constitutes the firstsignal processing stage and is informationally linked to the Houghtransformation unit 104. The Hough transformation unit 104 has a delayfilter 106, which can comprise at least one, however, advantageously aplurality of delay elements 108 a, 108 b, 108 c, 110 a, 110 b, and 110c. The delay elements 108 a to 108 c and 110 a to 110 c of the delayfilter 106 are typically arranged as a matrix, thus, in columns 108 and110 and lines a to c and signaling linked to each other. According tothe embodiment in FIG. 2a , at least one of the delay elements 108 a to108 c and/or 110 a to 110 c has an adjustable delay time, heresymbolized by means of the “+/−” symbols. For activating the delayelements 108 a to 108 c and 110 a to 110 c and/or for controlling thesame, a separate control logic and/or control register (not shown) canbe provided. This control logic controls the delay time of theindividual delay elements 108 a to 108 c and/or 110 a to 110 c viaoptional switchable elements 109 a to 109 c and/or 111 a to 111 c, whiche.g. can comprise a multiplexer and a bypass. The Hough transformationunit 104 can comprise an additional configuration register (not shown)for the initial configuration of the individual delay elements 108 a to108 c and 110 a to 110 c.

The pre-processor 102 has the objective to process the individualsamples 112 a, 112 b, and 112 c in a way that they can be efficientlyprocessed by the Hough transformation unit 104. For this purpose, thepre-processor 102 receives the image data and/or the plurality ofsamples 112 a, 112 b, and 112 c and performs a pre-processing, e.g. inform of a rotation and/or in form of a reflection, in order to outputthe several versions (cf. 112 a and 112 a′) to the Hough transformationunit 104. The outputting can occur serially, if the Hough transformationunit 104 has a Hough core 106, or also parallel, if several Hough coresare provided. Thus, this means that according to the implementation, then versions of the image are either entirely parallel, semi-parallel(thus, only partly parallel) or serially outputted and processed. Thepreprocessing in the pre-processor 102, which serves the purpose todetect several similar patterns (rising and falling straight line) witha search pattern or a Hough core configuration, is explained in thefollowing by means of the first sample 112 a.

This sample can e.g. be rotated, e.g. about 90° in order to obtain therotated version 112 a′. This procedure of the rotation has referencesign 114. Thereby, the rotation can occur either about 90°, but alsoabout 180° or 270° or generally about 360°/n, whereby it should be notedthat depending on the downstream Hough transformation (cf. Houghtransformation unit 104), it may be very efficient to carry out only a90° rotation. These sub-aspects are addressed with reference to FIGS. 2band 2c . Furthermore, the image 112 a can also be reflected, in order toobtain the reflected version 112 a″. The procedure of reflecting has thereference sign 116. The reflecting 116 corresponds to a rearwardread-out of the memory. Based on the reflected version 112 a″ as well asbased on the rotated version 112 a′, a fourth version can be obtainedfrom a rotated and reflected version 112 a′″, either by carrying out theprocedure 114 or 116. On the basis of the reflection 116, then twosimilar patterns (e.g. rightwards opened semicircle and leftwards openedsemicircle) with the same Hough core configuration as subsequentlydescribed, are detected.

The Hough transformation unit 104 is configured in order to detect inthe versions 112 a or 112 a′ (or 112 a″ or 112 a′″) provided by thepre-processor 102 a predetermined searched pattern, as e.g. an ellipsisor a segment of an ellipsis, a circle or a segment of a circle, astraight line or a graben segment. For this, the filter arrangement isconfigured corresponding to the searched predetermined pattern.Depending on the respective configuration, some of the delay elements108 a to 108 c or 110 a to 110 c are activated or bypassed. Hence, whenapplying a film strip of the image 112 a or 112 a′ to be examined to thetransformation unit 104 some pixels are selectively delayed by the delayelements 108 a to 108 c, which corresponds to an intermediate storageand others are directly transmitted to the next column 110. Due to thisprocedure, then curved or inclined geometries are “straightened”.Depending on the loaded image data 112 a or 112 a′, and/or, to beprecise, depending on the image structure of the applied line of theimage 112 a or 112 a′, high column amounts occur in one of the columns108 or 110, whereas the column amounts in other columns are lower. Thecolumn amount is outputted via the column amount output 108 x or 110 x,whereby here optionally an addition element (not shown) for establishingthe column amount of each column 108 or 110 can be provided. With amaximum of one of the column amounts, a presence of a searched imagestructure or of a segment of the searched image structure or at least ofthe associated degree of accordance with the searched structure can beassumed. Thus, this means that per processing step, the film strip ismoved further about a pixel or about a column 108 or 110 so that withevery processing step by means of a starting histogram, it isrecognizable, whether one of the searched structures is detected or not,or if the probability for the presence of the searched structure iscorrespondingly high. In other words, this means that overriding athreshold value of the respective column amount of column 108 or 110,show the detection of a segment of the searched image structure, wherebyevery column 108 or 110 is associated to a searched pattern or acharacteristic of a searched pattern (e.g. angle of a straight line orradius of a circle). It should be noted here that for the respectivestructure, not only the respective delay element 110 a, 110 b, and 110 cof the respective line 110 is decisive, but in particular the previousdelay elements 108 a, 108 b, and 108 c in combination with thesubsequent delay elements 110 a, 110 b, and 110 c. Corresponding to thestate of the art, such structures or activations of delay elements orbypass are a priori predetermined.

Via the variable delay elements 108 a to 108 c or 110 a to 110 c (delayelements), the searched characteristic (thus, e.g. the radius or theincrease) can be adjusted during ongoing operation. As the individualcolumns 108 and 110 are linked to each other, a change of the entirefilter characteristic of the filter 106 occurs during adjusting thedelay time of one of the delay elements 108 a to 108 c or 110 a to 110c. Due to the flexible adjustment of the filter characteristic of thefilter 106 of the Hough transformation unit 104, it is possible toadjust the transformation core 106 during the runtime so that e.g.dynamic image contents, as e.g. for small and large pupils can becollected and tracked with the same Hough core 106. In FIG. 3c , it isreferred to the exact implementation on how the delay time can beadjusted. In order to then enable the Hough processor 100 or thetransformation unit 104 having more flexibility, advantageously alldelay elements 108 a, 108 b, 108 c, 110 a, 110 b and/or 110 c (or atleast one of the mentioned) are carried out with a variable ordiscretely switchable delay time so that during the ongoing operation,it can be switched between the different patterns to be detected orbetween the different characteristics of the patterns to be detected.

According to further embodiments, the size of the shown Hough core 104is configurable (either during operation or previously) so that, thus,additional Hough cells can be activated or deactivated.

According to further embodiments, the transformation unit 104 can beconnected to means for adjusting the same or, to be precise, foradjusting the individual delay elements 108 a to 108 c and 110 a to 110c, as e.g. with a controller (not shown). The controller is e.g.arranged in a downstream processing unit and is configured in order toadjust the delay characteristic of the filter 106, if a pattern cannotbe recognized, or if the recognition is not sufficiently well (lowaccordance of the image content with the searched pattern of thepresence of the searched patterns). With reference to FIG. 5a , it isreferred to this controller.

The above mentioned embodiment has the advantage that it is easily andflexibly to be realized and that it is particularly able to beimplemented on an FPGA (Field Programmable Gate Array). The backgroundhereto is that the above described parallel Hough transformation getsalong without regression and is so to say entirely parallelized.Therefore, the further embodiments relate to FPGAs, which at least havethe Hough transformation unit 104 and/or the preprocessor 102. With animplementation of the above described device to an FPGA, e.g. a XILINXSpartan 3A DSP, a very high frame rate of e.g. 60 FPS with a resolutionof 640×480 could be achieved by using a frequency at 96 MHz, as due tothe above described structure 104 with a plurality of columns 108 and110, a parallel processing or a so-called parallel Hough transformationis possible.

FIGS. 2a and 2b show the pre-processor 102, which serves thepre-processing of the video data stream 112 with the frames 112 a, 112b, and 112 c. The pre-processor 102 is configured in order to receivethe samples 112 as binary edge images or even as gradient images and tocarry out on the basis of the same the rotation 114 or the reflection116, in order to obtain the four versions 112 a, 112 a′, 112 a″, and 112a′″. To this, the background is that typically the parallel Houghtransformation, as carried out by the Hough transformation unit, isbased on two or four respectively pre-processed, e.g. about 90° shiftedversions of an image 112 a. As shown in FIG. 2b , initially, a 90°rotation (112 a to 112 a′) occurs, before the two versions 112 a and 112a′ are horizontally reflected (cf. 112 a to 112 a″ and 112 a′ to 112a′″). In order to carry out the reflection 116 and/or the rotation 114,the pre-processor has in the corresponding embodiments an internal orexternal storage, which serves the charging of the received image data112.

The processing of rotating 114 and/or reflecting 116 of thepre-processor 102 depends on the downstream Hough transformation, thenumber of the parallel Hough cores (parallelizing degree) and theconfiguration of the same, as it is described in particular withreference to FIG. 2c . Insofar, the pre-processor 102 can be configuredin order to output the pre-processed video stream according to theparallelizing degree of the downstream Hough transformation unit 104corresponding to one of the three following constellations via theoutput 126:

100% parallelizing: simultaneous output of four video data streams,namely one non-rotated and non-reflected version 112 a, one about 90°rotated version 112 a′, and a respectively reflected version 112 a″ and112 a′″.

50% parallelizing: output of two video data streams, namely non-rotated112 a and about 90% reflected 112 a′ in a first step and output of therespectively reflected variants 112 a″ and 112 a′″ in a second step. 25%parallelizing: respective output of one video data stream, namelynon-rotated 112 a, about 90° rotated 112 a′, reflected 112 a″, andreflected and rotated 112 a′″, sequentially.

Alternatively to the above variant, it would also be conceivable thatbased on the first version, three further versions solely by rotation,thus, e.g. by rotation about 90°, 180°, and 270°, are established, onthe basis of which the Hough transformation is performed.

According to further embodiments, the pre-processor 102 can beconfigured in order to carry out further image processing steps, as e.g.an up-sampling. Additionally, it would also be possible that thepre-processor creates the gradient image. For the case that the gradientimage creation will be part of the image pre-processing, the grey-valueimage (initial image) could be rotated in the FPGA.

FIG. 2c shows two Hough core configurations 128 and 130, e.g. for twoparallel 31×31 Hough cores, configured in order to recognize a straightline or a straight section. Furthermore, a unit circle 132 is applied inorder to illustrate in which angle segment, the detection is possible.It should be noted at this point that the Hough core configuration 128and 130 is to be respectively seen in a way that the white dotsillustrate the delay elements. The Hough core configuration 128corresponds to a so-called type 1 Hough core, whereas the Hough coreconfiguration 120 corresponds to a so-called type 2 Hough core. Asderivable from the comparison of the two Hough core configurations 128and 130, the one constitutes the inverse of the other one. With thefirst Hough core configuration 128, a straight line in the segment 1between 3π/4 and π/2 can be detected, whereas a straight line in thesegment 3π/2 und 5π/4 (segment 2) is detectable by means of the Houghcore configuration 130. In order enable a detection in the furthersegments, as described above, the Hough core configuration 128 and 130is applied to the rotated version of the respective image. Consequently,by means of the Hough core configuration 128, the segment 1r between π/4and zero and by means of the Hough core configuration 130, the segment2r between π and 3π/4 can be collected.

Alternatively, when using only one Hough core (e.g. type 1 Hough core),a rotation of the image once about 90°, once about 180° and once about270° can be useful, in order to collect the above described variants ofthe straight line alignment. On the other hand, due to the flexibility,during the configuration of the Hough core, only one Hough core type canbe used, which is during ongoing operation reconfigured or regardingwhich the individual delay elements can be switched on or off in a way,that the Hough core corresponds to the inverted type. Thus, in otherwords, this means that when using the pre-processor 102 (in the 50%parallelizing operation) and the configurable Hough transformation unit104 with only one Hough core and with only one image rotation, theentire functionality can be displayed, which otherwise can only becovered by means of two parallel Hough cores. Insofar, it becomes clearthat the respective Hough core configuration or the selection of theHough core type depends on the pre-processing, which is carried out bythe pre-processor 102.

FIG. 3a shows a Hough core 104 with m columns 108, 110, 138, 140, 141,and 143 and n lines a, b, c, d, e, and f so that m×n cells are formed.The column 108, 110, 138, 140, 141, and 143 of the filter represents aspecific characteristic of the searched structure, e.g. for a specificcurve or a specific straight increase.

Every cell comprises a delay element, which is adjustable with respectto the delay time, whereby in this embodiment, the adjustment mechanismis realized due to the fact that respectively a switchable delay elementwith a bypass is provided. In the following, with reference to FIG. 3b ,the construction of all cells is representatively described. The cell(108 a) from FIG. 3b comprises the delay element 142, a remotecontrollable switch 144, as e.g. a multiplexer, and a bypass 146. Bymeans of the remote controllable switch 144, the line signal either cantransferred via the delay element 142 or it can be lead undelayed to theintersection 148. The intersection 148 is on the one hand connected tothe amount element 150 for the column (e.g. 108), whereby on the otherhand, via this intersection 148, also the next cell (e.g. 110 a) isconnected.

The multiplexer 144 is configured via a so-called configuration register160 (cf. FIG. 3a ). It should be noted at this point that the referencesign 160 shown here only relates to a part of the configuration register160, which is directly coupled to the multiplexer 144. The element ofthe configuration register 160 is configured in order to control themultiplexer 144 and receives thereto via a first information input 160a, a configuration information, which originates e.g. from aconfiguration matrix, which is stored in the FPGA internal BRAM 163.This configuration information can be a column-by-column bit string andrelates to the configuration of several (also during transformation) ofthe configured delaying cells (142+144). Therefore, the configurationinformation can be furthermore transmitted via the output 160 b. As thereconfiguration is not possible at any point in time of the operation,the configuration register 160 or the cell of the configuration register160 receives a so-called enabler signal via a further signal input 160c, by means of which the reconfiguration is started. Background to thisis that the reconfiguration of the Hough core needs a certain time,which depends on the number of delay elements or in particular on thesize of a column. Thereby, for every column element, a clock cycle isassociated and a latency of few clock cycles occurs due to the BRAM 163or the configuration logic 160. The total latency for thereconfiguration is typically negligible for video-based imageprocessing. It is assumed that in the present embodiment, the video datastreams recorded with a CMOS sensor have a horizontal and verticalblanking, whereby the horizontal blanking or the horizontal blankingtime can be used for the reconfiguration. Due to this context, the sizeof the Hough core structure implemented in the FPGA, predetermines themaximum size for the Hough core configuration. If e.g. smallerconfigurations are used, these are vertically centered and aligned inhorizontal direction to column 1 of the Hough core structure. Non-usedelements of the Hough core structure are all occupied with activateddelay elements.

The evaluation of the data streams processed in this way with theindividual delay elements (142+144) occurs column-by-column. For this,it is summed-up column-by-column, in order to detect a local amountmaximum, which displays a recognized searched structure. The summationper column 108, 110, 138, 140, 141, and 143 serves to determine a value,which is representative for the degree of accordance with the searchedstructure for one of the characteristic of the structure, assigned tothe respective column. In order to determine the local maxima of thecolumn amounts, per column 108, 110, 138, 140, 141, or 143, so-calledcomparer 108 v, 110 v, 138 v, 140 v, 141 v, or 143 v are provided, whichare connected to the respective amount elements 150. Optionally, betweenthe individual comparers 108 v, 110 v, 138 v, 140 v, 141 v, 143 v of thedifferent column 108, 110, 138, 140, 141, or 143, also further delayelements 153 can be provided, which serve to compare the column amountsof adjacent columns. In detail, during pass-through of the filter, thecolumns 108, 110, 138, or 140 with the highest degree of accordance fora characteristic of the searched pattern is picked out of the filter.During detecting a local maximum of a column amount (comparisonprevious, subsequent column), the presence of a searched structure canbe assumed. Thus, the result of the comparison is a column number(possibly including column amount=degree of accordance), in which thelocal maximum had been recognized ore in which the characteristic of thesearched structure is found, e.g. column 138. Advantageously, the resultcomprises a so-called multi-dimensional Hough room, which comprises allrelevant parameters of the searched structure, as e.g. the kind of thepattern (e.g. straight line or half circle), degree of accordance of thepattern, characteristic of the structure (intensity of the curveregarding curve segments or increase and length regarding straight linesegments) and the position or orientation of the searched pattern. Inother words, this means that for each point in the Hough room the greyvalues of the corresponding structure are added in the image segment.Consequently, maxima are formed by means of which the searched structurein the Hough room can easily be located and lead back to the imagesegment.

The Hough core cell from FIG. 3b can have an optional pipeline delayelement 162 (pipeline-delay), which e.g. is arranged at the output ofthe cell and is configured in order to delay the by means of the delayelement 142 delayed signal as well as the by means of the bypass 146non-delayed signal.

As indicated with reference to FIG. 1, such a cell also can have a delayelement with a variability or a plurality of switched and bypassed delayelements so that the delay time is adjustable in several stages.Insofar, further implementations beyond the implementation of the Houghcore cell as shown in FIG. 3b would alternatively be conceivable.

In the following, an application of the above described device within animage processing system 1000 is explained with reference to FIG. 5a .FIG. 5a shows an FPGA implemented image processor 10 a with apre-processor 102 and a Hough transformation unit 104. Prior to thepreprocessor 102, furthermore, an input stage 12 may be implemented inthe image processor 10 a, which is configured in order to receive imagedata or image samples from a camera 14 a. For this, the input stage 12may e.g. comprise an image takeover intersection 12 a, a segmentationand edge detector 12 b and measures for the camera control 12 c. Themeasures for the camera control 12 c are connected to the imageintersection 12 a and the camera 14 and serve to control the factorslike intensification and/or illumination.

The image processor 10 a further comprises a so-called Hough featureextractor 16, which is configured in order to analyze themulti-dimensional Hough room, which is outputted by the Houghtransformation unit 104 and which includes all relevant information forthe pattern recognition, and on the basis of the analyzing results tooutput a compilation of all Hough features. In detail, a smoothing ofthe Hough feature rooms occurs here, i.e. a spatial smoothing by meansof a local filter or a thinning of the Hough room (rejection ofinformation being irrelevant for the pattern recognition). This thinningis carried out under consideration of the kind of the pattern and thecharacteristic of the structure so that non-maxima in the Houghprobability room are faded out. Furthermore, for the thinning, alsothreshold values can be defined so that e.g. minimally or maximallyadmissible characteristics of a structure, as e.g. a minimal or amaximal curve or a smallest or greatest increase can be previouslydetermined.

By means of threshold-based rejection, also a noise suppression in theHough probability room may occur.

The analytical retransformation of the parameters of all remainingpoints in the original image segment, results e.g. from the followingHough features: for the curved structure, position (x- andy-coordinates), appearance probability, radius and angle, whichindicates to which direction the arc is opened, can be transmitted. Fora straight line, parameters as position (x- and y-coordinates),appearance probability, angle, which indicates the increase of astraight line, and length of the representative straight segment can bedetermined. This thinned Hough room is outputted by the Hough featureextractor 16 or generally, by the image processor 10 a for theprocessing at a post-processing unit 18.

The post-processing unit may e.g. be realized as embedded processor andaccording to its application, may comprise different sub-units, whichare exemplarily explained in the following. The post-processing unit 18may comprise a Hough feature post-geometry-converter 202. This geometryconverter 202 is configured in order to analyze one or more predefinedsearched patterns, which are outputted by the Hough feature extractorand to output the geometry explaining parameters. Thus, the geometryconverter 202 may e.g. be configured in order to output on the basis ofthe detected Hough features geometry parameters, as e.g. first diameter,second diameter, shifting and position of the midpoint regarding anellipsis (pupil) or a circle. According to an advantageous embodiment,the geometry converter 202 serves to detect and select a pupil by meansof 2 to 3 Hough features (e.g. curves). Thereby, criteria, as e.g. thedegree of accordance with the searched structure or the Hough features,the curve of the Hough features or the predetermined pattern to bedetected, the position and the orientation of the Hough features areincluded. The selected Hough feature combinations are arranged, wherebyprimarily the arrangement according to the amount of the obtained Houghfeatures and in a second line, according to the degree of accordancewith the searched structure occurs. After the arrangement, the Houghfeature combination at this point is selected and therefrom, theellipsis is fitted, which most likely represents the pupil within thecamera image.

Furthermore, the post-processing unit 18 comprises an optionalcontroller 204, which is formed to return a control signal to the imageprocessor 10 a (cf. control channel 206) or, to be precise, return tothe Hough transformation unit 104, on the basis of which the filtercharacteristic of the filter 106 is adjustable. For the dynamicadjustment of the filter core 106, the controller 204 typically isconnected to the geometry converter 202 in order to analyze the geometryparameters of the recognized geometry and in order to track the Houghcore within defined borders in a way that a more precise recognition ofthe geometry is possible. This procedure is a successive one, which e.g.starts with the last Hough core configuration (size of the lastly usedHough core) and is tracked, as soon as the recognition 202 providesinsufficient results. To the above discussed example of the pupil orellipsis detection, thus, the controller can adjust the ellipsis size,which e.g. depends on the distance between the object to be recorded andthe camera 14 a, if the person belonging thereto approaches the camera14 a. The control of the filter characteristic hereby occurs on thebasis of the last adjustments and on the basis of the geometryparameters of the ellipsis.

According to further embodiments, the post-processing unit 18 may have aselective-adaptive data processor 300. The data processor has thepurpose to post-process outliers and dropouts within a data series inorder to e.g. carry out a smoothing of the data series. Therefore, theselective-adaptive data processor 300 is configured in order to receiveseveral sets of values, which are outputted by the geometry converter202, whereby every set is assigned to respective sample. The filterprocessor of the data processor 300 carries out a selection of values onthe basis of the several sets in a way that the data values ofimplausible sets (e.g. outliers or dropouts) are exchanged by internallydetermined data values (exchange values) and the data values of theremaining sets are further used unchanged. In detail, the data values ofplausible sets (not containing outliers or dropouts), are transmittedand the data values of implausible sets (containing outliers ordropouts) are exchanged by data values of a plausible set, e.g. theprevious data value or by an average from several previous data values.The resulting data series from transmitted values and probably fromexchange values, is thereby continuously smoothened. Thus, this meansthat an adaptive time smoothing of the data series (e.g. of a determinedellipsis midpoint coordinate), e.g. occurs according to the principle ofthe exponential smoothing, whereby dropouts and outliers of the dataseries to be smoothened (e.g. due to erroneous detection during thepupil detection) do not lead to fluctuations of the smoothened data. Indetail, the data processor may smoothen over the data value of the newlyreceived set, if it does not fall within the following criteria:

-   -   According to the associated degree of accordance, which is        quantified by one of the additional values of the set, with the        searched structure, it is a dropout of the data series.    -   According to the associated size parameters or geometry        parameters, it is a dropout, if e.g. the size of the actual        object deviates too strong from the previous object.    -   According to a comparison of the actual data value with the        threshold values, which had been determined based on the        previous data values, it is a dropout, if the actual data value        (e.g. the actual position value) is not between the threshold        values. An illustrative example for this is, if e.g. the actual        position coordinate (data value of the set) of an object        deviates too strong from the previously by the selective        adaptive data processor determined position coordinate.

If one of these criteria is fulfilled, furthermore, the previous valueis outputted or at least consulted for smoothing the actual value. Inorder to obtain a possibly little delay during the smoothing, optionallythe actual values are stronger rated than past values. Thus, duringapplying of an exponential smoothing, the actual value can be determinedby means of the following formula:

Actually smoothened value=actual value x smoothing coefficient+lastsmoothened value x (1−smoothing coefficient)

The smoothing coefficient is within defined borders dynamically adjustedto the tendency of the data to be smoothened, e.g. reduction of therather constant value developments or increase regarding inclining orfalling value developments. If in a long-term a greater leap occursregarding the geometry parameters to be smoothened (ellipsisparameters), the data processor and, thus, the smoothened valuedevelopment adjust to the new value. Generally, the selective adaptivedata processor 300 can also be configured by means of parameters, e.g.during initializing, whereby via these parameters, the smoothingbehavior, e.g. maximum period of dropouts or maximum smoothing factor,are determined.

Thus, the selective adaptive data processor 300 or generally, thepost-processing unit 18 may output plausible values with high accuracyof the position and geometry of a pattern to be recognized. For this,the post-processing unit has an intersection 18 a, via which optionallyalso external control commands may be received. If more data seriesshall be smoothened, it is also conceivable to use for every data seriesa separate selective adaptive data processor or to adjust the selectiveadaptive data processor in a way that per set of data values, differentdata series can be processed.

In the following, the above features of the selective adaptive dataprocessor 300 are generally described by means of a concrete embodiment:

The data processor 300 e.g. may have two or more inputs as well as oneoutput. One of the inputs (receives the data value) is provided for thedata series to be processed. The output is a smoothened series based onselected data. For the selection, further inputs (the additional valuesfor the more precise assessment of the data values are received) areconsulted and/or the data series itself. During processing within thedata processor 300, a change of the data series occurs, whereby it isdistinguished between the treatment of outliers and the treatment ofdropouts within the data series.

Outliers: during the selection, outliers are (within the data series tobe processed) arranged and exchanged by other (internally determined)values.

Dropouts: For the assessment of the quality of the data series to beprocessed, one or more further input signals (additional values) areconsulted. The assessment occurs by means of one or more thresholdvalues, whereby the data is divided into “high” and “low” quality. Datawith a low quality are assessed being dropouts and are exchanged byother (internally determined) values.

In the next step, e.g. a smoothing of the data series occurs (e.g.exponential smoothing of a time series). For the smoothing, the dataseries is consulted, which has been adjusted of dropouts and outliers.The smoothing may occur by a variable (adaptive) coefficient. Thesmoothing coefficient is adjusted to the difference of the level of thedata to be processed.

According to further embodiments, it is also possible that thepost-processing unit 18 comprises an image analyzer, as e.g. a 3D imageanalyzer 400. In case of the 3D image analyzer 400, with thepost-processing unit 18 also a further image collecting unit consistingof an image processor 10 b and a camera 14 can be provided. Thus, twocameras 14 a and 14 b as well as the image processors 10 a and 10 bestablish a stereoscopic camera arrangement, whereby advantageously theimage processor 10 b is identical with the image processor 10 a.

The 3D image analyzer 400 is configured in order to receive at least oneset of image data, which is determined on the basis of one first image(cf. camera 14 a), and a second set of image data, which is determinedon the basis of a second image (cf. camera 14 b), whereby the first andthe second image display a pattern from different perspectives and inorder to calculate on the basis of this a point of view or a 3D gazevector. For this, the 3D image analyzer 400 comprises a positioncalculator 404 and an alignment calculator 408. The position calculator404 is configured in order to calculate a position of the pattern withina three-dimensional room based on the first set, the second set and ageometric relation between the perspectives or the first and the secondcamera 14 a and 14 b. The alignment calculator 408 is configured inorder to calculate a 3D gaze vector, e.g. a gaze direction, according towhich the recognized pattern is aligned to within the three-dimensionalroom, whereby the calculation is based on the first set, the second setand the calculated position (cf. position calculator 404).

For this, it may be e.g. consulted a so-called 3D camera system model,which e.g. has stored in a configuration file all model parameters, asposition parameter, optical parameter (cf. camera 14 a and 14 b).

In the following, based on the example of the pupil recognition, now theentire functionality of the 3D image analyzer 400 is described. Themodel stored or loaded in the 3D image analyzer 400 comprises dataregarding the camera unit, i.e. regarding the camera sensor (e.g. pixelsize, sensor size, and resolution) and the used objective lenses (e.g.focal length and objective lens distortion), data or characteristics ofthe object to be recognized (e.g. characteristics of an eye) and dataregarding further relevant objects (e.g. a display in case of using thesystems 1000 as input device).

The 3D position calculator 404 calculates the eye position or the pupilmidpoint on the basis of the two or even several camera images (cf. 14 aand 14 b) by triangulation. For this, it is provided with 2D coordinatesof a point in the two camera images (cf. 14 a and 14 b) via the processchain from image processors 10 a and 10 b, geometry converter 202 andselective adaptive data processor 300. From the delivered 2Dcoordinates, for both cameras 10 a and 10 b, the rays of light arecalculated, which have displayed the 3D point as 2D point on the sensor,by means of the 3D camera model, in particular under consideration ofthe optical parameters. The point of the two straight lines with thelowest distance to each other (in the ideal case, the intersection ofthe straight lines) is assumed as being the position of the searched 3Dpoint. This 3D position together with an error measure describing theaccuracy of the delivered 2D coordinates in connection with the modelparameters, is either via the intersection 18 a outputted as the result,or is transmitted to the gaze direction calculator 408.

On the basis of the position within the 3D room, the gaze directioncalculator 408 can determine the gaze direction from two ellipsis-shapedprojections of the pupil to the camera sensors without calibrating andwithout knowing the distance between the eyes and the camera system. Forthis, the gaze direction calculator 408 uses besides the 3D positionparameters of the image sensor, the ellipsis parameter, which aredetermined by means of the geometry analyzer 202 and the positiondetermined by means of the position calculator 404. From the 3D positionof the pupil midpoint and the position of the image sensors, by rotationof the real camera units, virtual camera units are calculated, theoptical axis of which passes through the 3D pupil midpoint.Subsequently, respectively from the projections of the pupil on the realsensors, projections of the pupil on the virtual sensors are calculatedso that two virtual ellipses arise. From the parameters of the virtualellipsis, the two points of view of the eye on an arbitrarily parallelplane to the respective virtual sensor plane, may be calculated. Withthe four points of view and the 3D pupil midpoints, four gaze directionvectors can be calculated, thus, respectively two vectors per camera.From these four possible gaze direction vectors, exactly one of the onecamera is nearly identical to the one of the other camera. Bothidentical vectors indicate the searched gaze direction of the eye, whichis then outputted by the gaze direction calculator 404 via theintersection 18 a.

A particular advantage of this 3D calculation is that a contactless andentirely calibration-free determination of the 3D eye position of the 3Dgaze direction and the pupil size does not depend on the knowledge onthe position of the eye towards the camera is possible. An analyticdetermination of the 3D eye position and the 3D gaze direction underconsideration of a 3D room model enables an arbitrary number of cameras(greater 1) and an arbitrary camera position in the 3D room. A shortlatency time with the simultaneously high frame rate enables a real-timecapability of the described system 1000. Furthermore, also the so-calledtime regimes are fixed so that the time differences between successiveresults are constant.

According to an alternative variant, it is also possible to carry out agaze direction determination, as in explained in the following withreference to FIG. 5.

In the previous description regarding the “3D image analyzer”, whichincludes the method for the calibration-free eye-tracking, so far atleast two camera images from different perspectives may have been used.Regarding the calculation of the gaze direction, there is a position, atwhich per camera image exactly two possible gaze direction vectors aredetermined, whereby respectively the second vector corresponds to areflection of the first vector at the intersection line between cameraand the pupil midpoint. From both vectors, which result from the othercamera image, exactly one vector nearly corresponds to a calculatedvector from the first camera image. These corresponding vectors indicatethe gaze direction to be determined.

In order to be able to carry out the calibration-free eye-tracking alsowith a camera, the actual gaze direction vector (in the following “vb”)has to be selected from the two possible gaze direction vectors, in thefollowing “v1” and “v2), which are determined from the camera image.

This process is explained with reference to FIG. 5b . FIG. 5b shows thevisible part of the eyeball (green framed) with the pupil and the twopossible gaze directions v1 and v2.

For selecting the gaze direction “vb”, there are several possibilities,which may be used individually or in combination in order to select theactual gaze direction vector. Some of these possibilities (the listingis not final) are explained in the following, whereby it is assumed thatv1 and v2 (cf. FIG. 5a ) have already been determined at the point intime of this selection:

Accordingly, a first possibility (the white dermis around the iris) mayoccur in the camera image. 2 beams are defined (starting at the pupilmidpoint and being infinitely long), one in the direction of v1 and onein the direction of v2. Both beams are projected into the camera imageof the eye and run there from the pupil midpoint to the image edge,respectively. The beam distorting the pixel which belong fewer to thesclera, belongs to the actual gaze direction vector vb. The pixel of thesclera differ by their grey value from those of the adjacent iris andfrom those of the eyelids. This method reaches its limits, if the facebelonging to the captured eye is too far averted from the camera (thus,if the angle between the optical axis of the camera and theperpendicularly on the facial plane standing vector becomes too large).

According to a second possibility, an evaluation of the position of thepupil midpoint may occur during the eye opening. The position of thepupil midpoint within the visible part of the eyeball or during the eyeopening, may be used for the selection of the actual gaze directionvector. One possibility thereto is to define two beams (starting at thepupil midpoint and being infinitely long), one in direction of v1 andone in direction of v2. Both beams are projected into the camera imageof the eye and run there from the pupil midpoint to the image edge,respectively. Along both beams in the camera image, respectively thedistance between the pupil midpoint and the edge of the eye opening (inFIG. 5b green marked) is determined. The beam, for which the shorterdistance arises, belongs to the actual gaze direction vector. Thismethod reaches its limits, if the if the face belonging to the capturedeye is too far averted from the camera (thus, if the angle between theoptical axis of the camera and the perpendicularly on the facial planestanding vector becomes too large).

According to a third possibility, an evaluation of the position of thepupil midpoint may occur towards a reference pupil midpoint. Theposition of the pupil midpoint determined in the camera image within thevisible part of the eyeball or during the eye opening may be usedtogether with a reference pupil midpoint for selecting the actual gazedirection vector. One possibility for this is to define 2 beams(starting at the pupil midpoint and being infinitely long), one indirection of v1 and one in direction of v2. Both beams are projectedinto the camera image of the eye and run there from the pupil midpointto the edge of the image, respectively. The reference pupil midpointduring the eye opening corresponds to the pupil midpoint if the eyedirectly looks to the direction of the camera sensor center which isused for the image recording. The beam projected into the camera image,which has in the image the smallest distance to the reference pupilmidpoint, belongs to the actual gaze direction vector. For determiningthe reference pupil midpoint, there are several possibilities, fromwhich some are described in the following:

Possibility 1 (specific case of application): The reference pupilmidpoint arises from the determined pupil midpoint, in the case, inwhich the eye looks directly in the direction of the camera sensorcenter. This is given, if the pupil contour on the virtual sensor plane(cf. description regarding gaze direction calculation) characterizes acircle.

Possibility 2 (general case of application): As rough estimate of theposition of the reference pupil midpoint the focus of the surface of theeye opening may be used. This method of estimation reaches its limits,if the plane in which the face is lying, is not parallel to the sensorplane of the camera. This limitation may be compensated, if theinclination of the facial plane towards the camera sensor plane is known(e.g. by a previously performed determination of the head position andalignment) and this is used for correction of the position of theestimated reference pupil midpoint.

Possibility 3 (general case of application): If the 3D position of theeye midpoint is available, a straight line between the 3D eye midpointand the virtual sensor midpoint can be determined as well as itsintersection with the surface of the eyeball. The reference pupilmidpoint arises from the position of this intersection converted intothe camera image.

According to further embodiments, instead of FPGAs 10 a and 10 b, anASIC (application specific chip) can be used, which is particularlyrealizable at high quantities with very low costs. Summarized, however,it can be established that independent from the implementation of theHough processor 10 a and 10 b, a low energy consumption due to thehighly efficient processing and the associated low internal clockrequirement can be achieved.

Despite these features, the here used Hough processor or the methodcarried out on the Hough processor remains very robust and notsusceptible to failures. It should be noted at this point that the Houghprocessor 100 as shown in FIG. 1 can be used in various combinationswith different features, in particular presented regarding FIG. 5.

Applications of the Hough processor according to FIG. 1 are e.g. warningsystems for momentary nodding off or fatigue detectors as drivingassistance systems in the automobile sector (or generally forsecurity-relevant man-machine-interfaces). Thereby, by evaluation of theeyes (e.g. covering of the pupil as measure for the blink degree) andunder consideration of the points of view and the focus, specificfatigue pattern can be detected. Further, the Hough processor can beused regarding input devices or input interfaces for technical devices;whereby then the eye position and the gaze direction are used as inputparameters. a precise application would be the support of the user whenviewing screen contents, e.g. with highlighting of specific focusedareas. Such applications are in the field of assisted living, computergames, regarding optimizing of 3D visualizing by including the gazedirection, regarding market and media development or regardingophthalmological diagnostics and therapies of particular interest.

As already indicated above, the implementation of the above presentedmethod does not depend on the platform so that the above presentedmethod can also be performed on other units, as e.g. a PC. Thus, afurther embodiment relates to a method for the Hough processing with thesteps of processing a majority of samples, which respectively have animage by using a pre-processor, whereby the image of the respectivesample is rotated and/or reflected so that a majority of versions of theimage of the respective sample for each sample is outputted and of thecollection of predetermined patterns in a majority of samples on thebasis of the majority of versions by using a Hough transformation unit,which has a delay filter with a filter characteristic being dependent onthe selected predetermined set of patterns.

Even if in the above explanations in connection with the adjustablecharacteristic, it was referred to a filter characteristic, it should benoted at this point that according to further embodiments, theadjustable characteristic may also relate to the post-processingcharacteristic (curve or distortion characteristic) regarding a fast 2Dcorrelation. This implementation is explained with reference to FIG. 4ato FIG. 4 d.

FIG. 4a shows a processing chain 1000 of a fast 2D correlation. Theprocessing chain of the 2D correlation comprises at least the functionblocks 1105 for the 2D curve and 1110 for the merging. The procedureregarding the 2D curve is illustrated in FIG. 4b . FIG. 4b shows theexemplary compilation at templates. By means of FIG. 4c in combinationwith FIG. 4d , it becomes obvious, how a Hough feature can be extractedon the basis of this processing chain 1000. FIG. 4c exemplarily showsthe pixel-wise correlation with n templates (in this case e.g. forstraight lines with different increase) for the recognition of theellipsis 1115, while FIG. 4d shows the result of the pixel-wisecorrelation, whereby typically via the n result images still a maximumsearch occurs. Every result image contains one Hough feature per pixel.In the following, this Hough processing is described in the overallcontext.

Contrary to the implementation with a delay filter with adjustablecharacteristic (implementation optimized for parallel FPGA structures),regarding the here outlined Hough processing, which in particular ispredestined for a PC-based implementation, a part of the processingwould be exchanged by another approach.

So far, it was the fact that quasi every column of the delay filterrepresents a searched structure (e.g. straight line segments ofdifferent increase). With passing the filter, the column number with thehighest amount value is decisive. Thereby, the column number representsa characteristic of the searched structure and the amount valueindicates a measure for the accordance with the searched structure.

Regarding the PC-based implementation, the delay filter is exchanged byfast 2D correlation. The previous delay filter is to be formed accordingto the size in the position n of characteristics of a specific pattern.This n characteristics are stored as template in the storage.Subsequently, the pre-processed image (e.g. binary edge image orgradient image) is passed pixel-wise. At every pixel position,respectively all stored templates with the subjacent image content(corresponding to the post-processing characteristic) are synchronized(i.e. the environment of the pixel position (in size of the templates)is evaluated). This procedure is referred to as correlation in thedigital image processing. Thus, for every template a correlation valueis obtained—i.e. a measure for the accordance—with the subjacent imagecontent. Thus, the latter correspond to the column amounts form theprevious delay filter. Now, decision is made (per pixel) for thetemplate with the highest correlation value and its template number ismemorized (the template number describes the characteristic of thesearched structure, e.g. increase of the straight line segment).

Thus, per pixel a correlation value and a template number is obtained.Thereby, a Hough feature, as already outlined, may be entirelydescribed.

It should be further noted that the correlation of the individualtemplates with the image content may be carried out in the local area aswell as in the frequency area. This means that the initial image firstof all is correlated with respectively all n templates. N result imagesare obtained. If these result images are put one above the other (likein a cuboid), the highest correlation value per pixel would be searched(via all planes). Thereby, individual planes then represent theindividual templates in the cuboid. As a result, again an individualimage is obtained, which then per pixel contains a correlation measureand a template number—thus, per pixel one Hough feature.

Even if the above aspects had been described in connection with the“pupil recognition”, the above outlined aspects are also usable forfurther applications. Here, for example, the application “warningsystems for momentary nodding off” is to be mentioned, to which in thefollowing it is referred to in detail.

The warning system for momentary nodding off is a system consisting atleast of an image collecting unit, an illumination unit, a processingunit and an acoustic and/or optical signaling unit. By evaluation of animage recorded by the user, the device is able to recognize beginningmomentary nodding off or fatigue or deflection of the user and to warnthe user.

The system can e.g. be developed in a form that a CMOS image sensor isused and the scene is illuminated in the infrared range. This has theadvantage that the device works independently from the environmentallight and, in particular does not blind the user. As processing unit, anembedded processor system is used, which executes a software code on thesubjacent operation system. The signaling unit currently consists of amulti-frequency buzzer and an RGB-LED.

The evaluation of the recorded image can occur in form of the fact thatin a first processing stage, a face and an eye detection and an eyeanalysis are performed with a classifier. This processing stage providesfirst indications for the alignment of the face, the eye position andthe degree of the blink reflex.

Based on this, in the subsequent step, a model-based eye preciseanalysis is carried out. An eye model used therefor can e.g. consist of:a pupil and/or iris position, a pupil and/or iris size, a description ofthe eyelids and the eye edge points. Thereby, it is sufficient, if atevery point in time, some of these components are found and evaluated.The individual components may also be tracked via several images so thatthey have not to be completely searched again in every image.

The previously described Hough features can be used in order to carryout the face detection or the eye detection or the eye analysis or theeye precise analysis. The previously described 2D image analyzer can beused for the face detection or the eye detection or the eye analysis.For the smoothing of the determined result values or intermediateresults or value developments during the face detection or eye detectionor eye analysis, the described adaptive selective data processor can beused.

A chronological evaluation of the degree of the blink reflex and/or theresults of the eye precise analysis, can be used for determining themomentary nodding of or the fatigue or deflection of the user.Additionally, also the calibration-free gaze direction determination asdescribed in connection with the 3D image analyzer can be used in orderobtain better results for the determination of the momentary nodding offor the fatigue or deflection of the user. In order to stabilize theseresults, moreover, the adaptive selective data processor can be used.

The procedure described in the embodiment “momentary nodding off warningsystem” for the determination of the eye position can also be used forthe determination of an arbitrarily other defined 2D position, as e.g.the nose position, or nasal root position within a face.

When using one set of information from one image and a further set ofinformation, this position can also be determined in the 3D room,whereby the further set of information can be generated from an image ofa further camera or by evaluation of relations between objects in thefirst camera image.

According to an embodiment, the Hough processor in the stage of initialimage can comprise a unit for the camera control.

Although some aspects have been described in connection with a device,it is understood that these aspects also constitute a description of therespective method so that a block or a component of a device is also tobe understood as being a respective method step or a feature of a methodstep. Analogous thereto, aspects which had been described in connectionwith or as being a method step, also constitute a description of arespective block or detail or feature of the respective device. Some orall method steps may be carried out by an apparatus (by using a hardwareapparatus), as e.g. a microprocessor, of a programmable computer or anelectronic switch. Regarding some embodiments, some or more of theimportant method steps can be carried out by such an apparatus.

According to specific implementation requirements, embodiments ofinvention may be implemented into hardware or software. Theimplementation may be carried out by using a digital storage medium, ase.g. a Floppy Disc, a DVD, a Blu-ray Disc, a CD, a ROM, a PROM, amEPROM, an EEPROM, or a FLASH memory, a hard disc or any other magneticor optical storage, on which electronically readable control signals arestored, which collaborate with a programmable computer system in a waythat the respective method is carried out. Therefore, the digitalstorage medium may be computer readable.

Some embodiments according to the invention, thus, comprise a datacarrier having electronically readable control signals, which are ableto collaborate with a programmable computer system in a way that one ofthe herein described methods is carried out.

Generally, embodiments of the present invention can be implemented ascomputer program product with a program code, whereby the program codeis effective in order to carry out one of the methods, if the computerprogram product runs on a computer.

The program code may e.g. be stored on a machine-readable carrier.

Further embodiments comprise the computer program for the execution ofone of the methods described herein, whereby the computer program isstored on a machine-readable carrier.

In other words, thus, one embodiment of the method according to theinvention is a computer program having a program code for the executionof one of the methods defined herein, if the computer program runs on acomputer.

Thus, a further embodiment of the inventive method is a data carrier (ora digital storage medium or a computer-readable medium) onto which isrecorded the computer program for performing any of the methodsdescribed herein.

A further embodiment of the method according to the invention, thus, isa data stream or a sequence of signals, which constitute the computerprogram for carrying out one of the herein defined methods. The datastream or the sequence of signals can e.g. be configured in order to betransferred via a data communication connection, e.g. via the Internet.

A further embodiment comprises a processing unit, e.g. a computer or aprogrammable logic component, which is configured or adjusted in orderto carry out one of the herein defined methods.

A further embodiment comprises a computer, on which the computer programfor executing one of the herein defined method is installed.

A further embodiment according to the invention comprises a device or asystem, which are designed in order to transmit a computer program forexecuting at least one of the herein defined methods to a recipient. Thetransmission may e.g. occur electronically or optically. The recipientmay be a computer, a mobile device, a storage device, or a similardevice. The device or the system can e.g. comprise a file server for thetransmission of the computer program to the recipient.

Regarding some embodiments, a programmable logic component (e.g. a fieldprogrammable gate array, an FPGA) may be used in order to execute someor all functionalities of the herein defined methods. Regarding someembodiments, a field-programmable gate array can collaborate with amicroprocessor, in order to execute one of the herein defined methods.Generally, regarding some embodiments, the methods are executed by anarbitrary hardware device. This can be a universally applicable hardwareas a computer processor (CPU) or a hardware specific for the method, ase.g. an ASIC.

In the following, the above described inventions or aspects of theinventions are described from two further perspectives in other words:

Integrated Eye-Tracker

The integrated eye-tracker comprises a compilation of FPGA-optimizedalgorithms, which are suitable to extract (ellipsis) features (Houghfeatures) by means of a parallel Hough transformation from a camera liveimage. Thereafter, by evaluating the extracted features, the pupilellipsis can be determined. When using several cameras with a positionand alignment known to each other, the 3D position of the pupil midpointas well as the 3D gaze direction and the pupil diameter can bedetermined. For the calculation, the position and form of the ellipsisin the camera images are consulted. Calibration of the system for therespective user is not required as well as knowledge of the distancebetween the cameras and the analyzed eye. The used image processingalgorithms are in particular characterized in that they are optimizedfor the processing on an FPGA (field programmable gate array). Thealgorithms enable a very fast image processing with a constant refreshrate, minimum latency periods and minimum resource consumption in theFPGA. Thus, these modules are predestined for time-, latency, andsecurity-critical applications (e.g. driving assistance systems),medical diagnostic systems (e.g. perimeters) as well as application forhuman machine interfaces (e.g. mobile devices), which involve a smallconstruction volume.

Objective Technical Problem

-   -   Robust detection of 3D eye positions and 3D gaze directions in        the 3D room in several (live) camera images as well as detection        of the pupil size    -   Very short reaction period (or processing time)    -   Small construction    -   Autonomous functionality (independent from the PC) by integrated        solution

State of the Art

-   -   Eye-tracker systems        -   Steffen Markert: gaze direction determination of the human            eye in real time (diploma thesis and patent DE 10 2004 046            617 A1)        -   Andrew T. Duchowski: Eye Tracking Methodology: Theory and            Practice    -   Parallel Hough Transformation        -   Johannes Katzmann: A real time implementation for the            ellipsis Hough transformation (diploma thesis and patent DE            10 2005 047 160 B4)        -   Christian Holland-Nell: Implementation of a pupil detection            algorithm based on the Hough transformation for circles            (diploma thesis and patent DE 10 2005 047 160 B4)

Disadvantages of the Actual State of the Art

-   -   Eye-tracker systems        -   Disadvantages:            -   Eye-tracking systems generally involve a (complex)                calibration prior to use            -   The system according to Markert (patent DE 10 2004 046                617 A1) is calibration-free, however works only under                certain conditions:                -   1. Distance between camera and pupil midpoint has to                    be known and on file                -   2. The method only works for the case that the 3D                    pupil midpoint lies within the optical axes of the                    cameras            -   The overall processing is optimized for PC hardware and,                thus, is also subject to their disadvantages (no fixed                time regime is possible during the processing)            -   Efficient systems may be used, as the algorithms have a                very high resource consumption            -   Long processing period and, thus, long delay periods                until the result is available (partly dependent on the                image size to be evaluated)    -   Parallel Hough Transformation        -   Disadvantages:            -   Only binary edge images can be transformed            -   Transformation only provides a binary result related to                an image coordinate (position of the structure was                found, but not: hit probability and further structure                features)            -   No flexible adjustment of the transformation core during                the ongoing operation and, thus, only insufficient                suitability for dynamic image contents (e.g. small and                big pupils)            -   Reconfiguration of the transformation core to other                structures during operation is not possible and, thus                limited suitability for object recognition

Implementation

The overall system determines from two or more camera images, in whichthe same eye is displayed, respectively a list of multi-dimensionalHough features and respectively calculates on their basis the positionand form of the pupil ellipsis. From the parameters of these twoellipses as well as solely from the position and alignment of the camerato each other, the 3D position of the pupil midpoint as well as the 3Dgaze direction and the pupil diameter can be determined entirelycalibration-free. As hardware platform, a combination of at least twoimage sensors, FPGA and/or downstream microprocessor system is used(without the mandatory need of a PCI).

“Hough preprocessing”, “Parallel Hough transform”, “Hough featureextractor”, “Hough feature to ellipse converter”, “Core-size control”,“Temporal smart smoothing filter”, “3D camera system model”, “3Dposition calculation” and “3D gaze direction calculation” relate toindividual function modules of the integrated eye tracker. They fallinto line of the image processing chain of the integrated eye-tracker asfollows:

FIG. 6 shows a block diagram of the individual function modules in theintegrated eye-tracker. The block diagram shows the individualprocessing stages of the integrated eye-tracker. In the following, adetailed description of the modules is presented.

-   -   “Hough pre-processing”        -   Function            -   Up-sampling of a video stream for the module “Parallel                Hough Transform”, in particular by image rotation and                up-sampling of the image to be transformed according to                the parallelizing degree of the module “Parallel Hough                Transform”        -   Input            -   Binary edge image or gradient image        -   Output            -   According to the parallelizing degree of the subsequent                module, one or more video streams with up-sampled pixel                data from the input        -   Detailed description            -   Based on the principle, the parallel Hough                transformation can be applied to the image content from                four about respectively 90° distorted main directions            -   For this, in the pre-processing, an image rotation of                about 90° occurs            -   The two remaining directions are covered by the fact                that respectively the rotated and the non-rotated image                are horizontally reflected (by reverse read-out of the                image matrix filed in the storage)            -   According to the parallelizing degree of the module, the                following three constellations arise for the output:                -   100% parallelizing: simultaneous output of four                    video data streams: about 90° rotated, non-rotated                    as well as respectively reflected                -   50% parallelizing: output of two video data streams:                    about 90° rotated and non-rotated, the output of the                    respectively reflected variations occurs                    sequentially                -   25% parallelizing: output of a video data stream:                    about 90° rotated and non-rotated and respectively                    their reflected variations are outputted                    sequentially    -   “Parallel Hough transform”        -   Function            -   Parallel recognition of simple patterns (straight lines                with different sizes and increases and curves with                different radii and orientations) and their appearance                probability in a binary edge or gradient image        -   Input            -   For the parallel Hough Transformation up-sampled edge or                gradient image (output of the “Hough preprocessing”                module)        -   Output            -   Multi-dimensional Hough room containing all relevant                parameters of the searched structure        -   Detailed description            -   Processing of the input by a complex delay-based local                filter, which has a defined “passing direction” for                pixel data and is characterized by the following                features:                -   Filter core with variable size consisting of delay                    elements                -   For the adaptive adjustment of the filter to the                    searched patterns, delay elements can be switched on                    and off during the operation                -   Every column of the filter represents a specific                    characteristic of the searched structure (curve or                    straight line increase)                -   Summation via the filter columns provides appearance                    probability for the characteristic of the structure                    represented by the respective column                -   When passing the filter, the column with the highest                    appearance probability for a characteristic of the                    searched pattern is outputted            -   For every image pixel, the filter provides one point in                the Hough room, which contains the following                information:                -   Kind of the pattern (e.g. straight line or half                    circle)                -   Appearance probability for the pattern                -   Characteristic of the structure (intensity of the                    curve or for straight lines: increase and length)                -   Position or orientation of the structure in the                    image            -   As transformation result, a multi-dimensional image                arises, which is in the following referred to as Hough                room.    -   “Hough feature extractor”        -   Function            -   Extraction of features from the Hough room containing                relevant information for the pattern recognition        -   Input            -   Multi-dimensional Hough room (output of the “parallel                Hough transform” module)        -   Output            -   List of Hough features containing relevant information                for the pattern recognition        -   Detailed description            -   Smoothing of the Hough feature rooms (spatial correction                by means of local filtering)            -   “Thinning” of the Hough room (suppression of                non-relevant information for the pattern recognition) by                a modified “non-maximum-suppression”:                -   Fading out of points non-relevant for the processing                    (“non-maxima” in the Hough probability room) by                    considering the kind of the pattern and the                    characteristic of the structure                -   Further thinning of the Hough room points by means                    of suitable thresholds:                -    Noise suppression by threshold value in the Hough                    probability room                -    Indication of an interval for minimum and maximum                    admissible characteristic of the structure (e.g.                    minimum/maximum curve regarding curved structures or                    lowest/highest increase regarding straight lines)            -   Analytical retransformation of the parameters of all                remaining points in the original image scope results in                the following Hough features:                -   Curved structures with the parameters:                -    Position (x- and y-image coordinates)                -    Appearance probability of the Hough features                -    Radius of the arc                -    Angle indicating in which direction the arc is                    opened                -   Straight lines with the parameters:                -    Position (x- and y-image coordinates)                -    Appearance probability of the Hough features                -    Angle indicating the increase of the straight line                -    Length of the represented straight line segment    -   “Hough feature to ellipse converter”        -   Function            -   Selection of the 3 to 4 Hough features (curves), which                describe with the highest probability the pupil edge                (ellipsis) in the image and settling to an ellipsis        -   Input            -   List of all detected Hough features (curves) in a camera                image        -   Output            -   Parameter of the ellipsis representing with the highest                probability the pupil        -   Detailed description            -   From the list of all Hough features (curves),                combinations of 3 to 4 Hough features are formed, which                due to their parameters can describe the horizontal and                vertical extreme points of an            -   Thereby, the following criteria have an influence on the                selection of the Hough features:                -   Scores (probabilities) of the Hough features                -   Curve of the Hough features                -   Position and orientation of the Hough features to                    each other            -   The selected Hough feature combinations are arranged:                -   Primarily according to the number of the contained                    Hough features                -   Secondary according to combined probability of the                    contained Hough features            -   After arranging, the Hough feature combination being in                the first place is selected and the ellipsis, which                represents most probably the pupil in the camera image,                is fitted    -   “Core-size control”        -   Function            -   Dynamic adjustment of the filter core (Hough core) of                the parallel Hough transformation to the actual ellipsis                size        -   Input            -   Last used Hough core size            -   Parameters of the ellipsis, which represents the pupil                in the corresponding camera image        -   Output            -   Updated Hough core size        -   Detailed description            -   Dependent on the size (length of the half axes) of the                ellipsis calculated by the “Hough feature to ellipse                converter”, the Hough core size is tracked in order to                increase the accuracy of the Hough transformation                results during the detection of the extreme points    -   “Temporal smart smoothing filter”        -   Function            -   Adaptive simultaneous smoothing of the data series (e.g.                of a determined ellipsis midpoint coordinate) according                to the principle of the exponential smoothing, whereby                the dropouts or extreme outliers within the data series                to be smoothened do NOT lead to fluctuations of the                smoothened data        -   Input            -   At every activation time of the module, respectively one                value of the data series and the associated quality                criteria (e.g. appearance probability of a fitted                ellipsis)        -   Output            -   Smoothened data value (e.g. ellipsis midpoint                coordinate)        -   Detailed description            -   Via a set of filter parameters, with initializing the                filter, its behavior can be determined            -   The actual input value is used for the smoothing, if it                does not fall within one of the following categories:                -   Corresponding to the associated appearance                    probability, it is a dropout in the data series                -   Corresponding to the associated ellipsis parameters,                    it is an outlier                -    If the size of the actual ellipsis differs to much                    from the size of the previous ellipsis                -    With a too large difference of the actual position                    towards the last position of the ellipsis            -   If one of these criteria is fulfilled, furthermore, the                previously determined value is outputted, otherwise, the                current value for smoothing is consulted            -   In order to obtain a possibly low delay during                smoothing, current values are stronger rated than past                ones:                -   Currently smoothened value=current value*smoothing                    coefficient+last smoothened value*(1−smoothing                    coefficient)                -   The smoothing coefficient is adjusted within defined                    borders dynamically to the tendency of the data to                    be smoothened:                -    Reduction with rather constant value development in                    the data series                -    Increase with increasing or decreasing value                    development in the data series            -   If in the long term a larger leap regarding the ellipsis                parameters to be smoothened occurs, the filter and,                thus, also the smoothened value development adjusts    -   “3D camera system model”        -   Function            -   Modeling of the 3D room, in which several cameras, the                user (or his/her eye) and possibly a screen are located        -   Input            -   Configuration file, containing the model parameters                (position parameter, optical parameters, amongst others)                of all models        -   Output            -   Provides a statistical framework and functions for the                calculations within this model        -   Detailed description            -   Modeling of the spatial position (position and rotation                angle) of all elements of the model as well as their                geometric (e.g. pixel size, sensor size, resolution) and                optical (e.g. focal length, objective distortion)                characteristics            -   The model comprises at this point in time the following                elements:                -   Camera units, consisting of:                -    Camera sensors                -    Objective lenses                -   Eyes                -   Display            -   Besides the characteristics of all elements of the                model, in particular the subsequently described                functions “3D position calculation” (for the calculation                of the eye position) and “3D gaze direction calculation”                (for the calculation of the gaze direction) are provided            -   By means of this model, inter alia the 3D line of sight                (consisting of the pupil midpoint and the gaze direction                vector (corrected corresponding to biology and                physiology of the human eye can be calculated            -   Optionally, also the point of view of a viewer on                another object of the 3D model (e.g. on a display) may                be calculated as well as the focused area of the viewer    -   “3D position calculation”        -   Function            -   Calculation of the spatial position (3D coordinates) of                a point, captured by two or more cameras (e.g. pupil                midpoint) by triangulation        -   Input            -   2D-coordinates of one point in two camera images        -   Output            -   3D-coordinates of the point            -   Error measure: describes the accuracy of the transferred                2D-coordinates in combination with the model parameters        -   Detailed description            -   From the transferred 2D-coordinates, by means of the “3D                camera system model” (in particular under consideration                of the optical parameters) for both cameras, the light                beams are calculated, which have displayed the 3D point                as 2D point on the sensors            -   These light beams are described as straight lines in the                3D room of the mode            -   The point of which both straight lines have the smallest                distance (in the ideal case, the intersection of the                straight lines), is assumed to be the searched 3D point    -   “3D gaze direction calculation”        -   Function            -   Determination of the gaze direction from two                ellipsis-shaped projections of the pupil to the camera                sensors without calibration and without knowledge of the                distance between eye and camera system        -   Input            -   3D position parameters of the image sensors            -   Ellipsis parameters of the pupil projected to both image                sensors            -   3D positions of the ellipsis midpoint on both image                sensors            -   3D position of the pupil midpoint        -   Output            -   3D gaze direction in vector and angle demonstration        -   Detailed description            -   From the 3D position of the pupil midpoint and the                position of the image sensors, by rotation of the real                camera units, virtual camera units are calculated, the                optical axis of which passes through the 3D pupil                midpoint            -   Subsequently, from the projections of the pupil to the                real sensor projections of the pupil, respectively the                virtual sensors are calculated, thus, so to speak, two                virtual ellipses arise            -   From the parameters of the virtual ellipses, for both                sensors, respectively two view points of the eye can be                calculated on a parallel plane being arbitrary parallel                to the respective sensor plane            -   With these four points of view and the 3D pupil                midpoint, four gaze direction vectors can be calculated                (respectively two vectors from the results of each                camera)            -   From these four gaze direction vectors, exactly one is                (nearly) identical with one of the one camera with one                of the other camera            -   Both identical vectors indicate the searched gaze                direction of the eye, which is then provided by the                module “3D gaze direction calculation” as result

4. a) Advantages

-   -   Contactless and completely calibration-free determination of the        3D eye positions, 3D gaze direction and pupil size independent        from the knowledge of the eye's position towards the cameras    -   Analytical determination of the 3D eye position and 3D gaze        direction (by including a 3D room model) enables an arbitrary        number of cameras (>2) and an arbitrary camera position in the        3D room    -   Measuring of the pupil projected to the camera and, thus, a        precise determination of the pupil size    -   High frame rates (e.g. 60 FPS @ 640×480 on one XILINX Spartan 3A        DSP @ 96 MHz) and short latency periods due to completely        parallel processing without recursion in the processing chain    -   Use of FPGA hardware und algorithms, which had been developed        for the parallel FPGA structures    -   Use of the Hough transformation (in the described adjusted form        for FPGA hardware) for the robust feature extraction for the        object recognition (here: features of the pupil ellipsis)    -   Algorithms for the post-processing of the Hough transformation        results are optimized on parallel processing in FPGAs    -   Fixed time regime (constant time difference between consecutive        results)    -   Minimum construction room, as completely integrated on a chip    -   Low energy consumption    -   Possibility for a direct porting of the processing to FPGA to an        ASIC very cost-effective solution with high quantities due to        exploitation of scaling effects

Workarounds

-   -   (partly) use of other algorithms (e.g. a different object        recognition method for the ellipsis detection

Possibility of Proof for Patent Infringement

-   -   Obviously, a patent infringement could exist, if the        corresponding product is a fast and completely calibration-free        eye-tracking system consisting of FPGA and microprocessor.    -   Proof of patent infringement by copying/cloning of the FPGA bit        file/network lists        -   Would be easily to prove, e.g. by check amounts        -   Further, bit files can be bound to a FPGA-ID the copying            would then only be possible by using of FPGA similar IDs    -   Proof of patent infringement by disassembling of the FPGA bit        file/network lists        -   Indications for patent infringements would possibly be            recognizable by disassembling of the respective FPGA bit            file/network list        -   The concrete proof could only hardly be rendered    -   Proof of patent infringement by “disassembling” of the processor        codes        -   Indications would be recognizable, concrete proof only            hardly possible

Application

-   -   In a (live-) camera image data stream, 3D eye positions and 3D        gaze directions are detected, which can be used for the        following applications:        -   Security-relevant fields            -   e.g. momentary nodding off warning system or fatigue                detectors as driving assistance system in the automotive                sector, by evaluation of the eyes (e.g. coverage of the                pupil as measure for the blink degree) and under                consideration of the points of view and the focus        -   Man-machine-interfaces            -   As input interfaces for technical devices (eye position                and gaze direction may be used as input parameters)            -   Support of the user when viewing screen contents (e.g.                highlighting of areas, which are viewed)            -   E.g.                -   in the field of Assisted Living                -   for computer games                -   gaze direction supported input for Head Mounted                    Devices                -   optimizing of 3D visualizations by including the                    gaze direction        -   Market and media development            -   E.g. assessing attractiveness of advertisement by                evaluating of the spatial gaze direction and the pint of                view of the test person        -   Ophthalmological diagnostic (e.g. objective perimetry) and            therapy

FPGA-Face Tracker

One aspect of the invention relates to an autonomous (PC-independent)system, which in particular uses FPGA-optimized algorithms and which issuitable to detect a face in a camera live image and its (spatial)position. The used algorithms are in particular characterized in thatthey are optimized for the processing on an FPGA (field programmablegate array) and compared to the existing methods, get along withoutrecursion in the processing. The algorithms allow a very fast imageprocessing with constant frame rate, minimum latency periods and minimumresource consumption in the FPGA. Thereby, these modules are predestinedfor a time/latency-/security-critical application (e.g. drivingassistance systems) or applications as human machine interfaces (e.g.for mobile devices), which involve a small construction volume.Moreover, by using a second camera, the spatial position of the user forspecific points in the image may be determined highly accurate,calibration-free and contactless.

Objective Technical Problem

Robust and hardware-based face detection in a (live) camera image

-   -   Detection of face and eye position in the 3D room by using a        stereoscopic camera system    -   Very short reaction period (or processing period)    -   Small construction    -   Autonomous functionality (independency from the PC) by        integrated solution

State of the Art

-   -   Literature:        -   Christian Küblbeck, Andreas Ernst: Face detection and            tracking in video sequences using the modified census            transformation        -   Paul Viola, Michael Jones: Robust Real-time Object Detection

Disadvantages of Current Face Tracker Systems

-   -   The overall processing is optimized for PC systems (more        general: general purpose processors) and, thus, is also subject        to their disadvantages (e.g. fixed time regime during processing        is not possible (example: dependent on the image content, e.g.        background, the tracking possibly takes a longer time))    -   Sequential processing; the initial image is successively brought        into different scaling stages (until the lowest scaling stage is        reached) and is searched respectively with a multi-stage        classifier regarding faces        -   Depending on how many scaling stages have to be calculated            or how many stages of the classifier have to be calculated,            the processing period varies until the result is available    -   In order to reach high frame rates, efficient systems may be        used (higher clock rates, under circumstances multi-score        systems), as the already to PC hardware optimized algorithms        despite have a very high resource consumption (in particular        regarding embedded processor systems)    -   Based on the detected face position, the classifiers provide        only inaccurate eye positions (the eyes' position—in particular        the pupil midpoint—is not analytically determined (or measured)        and is therefore subject to high inaccuracies)    -   The determined face and eye positions are only available within        the 2D image coordinates, not in 3D

Implementation

The overall system determines from a camera image (in which only oneface is displayed) the face position and determines by using thisposition, the positions of the pupil midpoints of the left and righteye. If two or more cameras with a known alignment to each other areused, these two points can be indicated for the three-dimensional room.Both determined eye positions may be further processed in systems, whichuse the “integrated eye-tracker”. The “parallel image scaler”, “parallelface finder”, “parallel eye analyzer”, “parallel pupil analyzer”,“temporal smart smoothing filter”, “3D camera system model” and “3Dposition calculation”relate to individual function modules of theoverall system (FPGA face tracker). They get in lane with the imageprocessing chain of FPGA face trackers as follows:

FIG. 7a shows a block diagram of the individual function modules in theFPGA face tracker 800. The function modules “3D camera system model” 802and “3D position calculation” 804 are mandatory for the face tracking,however, are used when using a stereoscopic camera system andcalculating suitable points on both cameras for the determination ofspatial positions (e.g. for determining the 3D head position duringcalculation of the 2D face midpoints in both camera images). The module“feature extraction (classification)” 806 of the FPGA face trackers isbased on the feature extraction and classification of Küblbeck/Ernst ofFraunhofer ITS (Erlangen, Germany) and uses an adjusted variant of itsclassification on the basis of census features.

The block diagram shows the individual processing stages of the FPGAface tracking system. In the following, a detailed description of themodules is presented.

-   -   “Parallel image scaler 702”        -   Function            -   Parallel calculation of the scaling stages of the                initial image and arrangement of the calculated scaling                stages in a new image matrix in order to allow the                subsequent image processing modules a simultaneous                analysis of all scaling stages

FIG. 7b shows the initial image 710 (original image) and result 712(downscaling image) of the parallel image scaler.

-   -   Input        -   Initial image 710 in original resolution    -   Output 712        -   New image matrix containing more scaled variants of the            initial image in an arrangement suitable for the subsequent            face tracking modules    -   Detailed description        -   Establishing an image pyramid by parallel calculation of            different scaling stages of the initial image        -   In order to guarantee a defined arrangement of the            previously calculated scaling stages within the target            matrix, a transformation of the image coordinates of the            respective scaling stages into the image coordinate system            of the target matrix occurs by means of various criteria:            -   Defined minimum distance between the scaling stages in                order to suppress a crosstalk of analysis results in                adjacent stages            -   Defined distance to the edges of the target matrix in                order to guarantee the analysis of faces partly                projecting from the image    -   “Parallel face finder 808”    -   Function        -   Detects a face from classification results of several            scaling stages, which are jointly arranged in a matrix. The            parallel face finder 808 is comparable with the finder from            FIG. 1, whereby the finder 706 comprises a general            functionality scope (recognition of further patterns, as            pupil recognition).

As shown in FIG. 7c , the result of the classification (rightwards)constitutes the input for the parallel face finder.

-   -   Input 712        -   Classified image matrix containing several scaling stages    -   Output        -   Position at which with highest probability a face is located            (under consideration of several criteria)    -   Detailed description        -   Noise suppression for limiting the classification results        -   Spatial correction of the classification results within the            scaling scales by means of a combination of local amount and            maximum filter        -   Orientation on the highest appearance probability for a face            optionally at the face size over and away of all scaling            stages        -   Spatial averaging of the result positions over and away of            selected scaling stages            -   Selection of the scaling stages included in the                averaging takes place under consideration of the                following criteria:                -   Difference of the midpoints of the selected face in                    the viewed scaling stages                -   Dynamically determined deviation of the highest                    result of the amount filter                -   Suppression of scaling stages without classification                    result        -   Threshold-based adjustment of the detection performance of            the “parallel face finder”    -   “Parallel eye analyzer 810”    -   Function        -   Detects parallel during the face detection the position of            the eyes in the corresponding face (this is above all            important for not ideally frontally captured and distorted            faces)    -   Input        -   Image matrix containing several scaling stages of the            initial image (from the “parallel image scaler” module) as            well as the respective current position, at which the            searched face with highest probability is located (from the            “parallel face finder” module)    -   Output        -   Position of the eyes and an associated probability value in            the currently detected face by the “parallel face finder”    -   Detailed description        -   Based on the down-scaled initial image, in its defined range            (eye range) within the face region provided by the “parallel            face finder”, the eye search for every eye is executed as            described in the following:            -   Defining the eye range from empirically determined                normal positions of the eyes within the face region.            -   With a specifically formed correlation-based local                filter, probabilities for the presence of an eye are                determined within the eye range (the eye in this image                segment is simplified described as a little dark surface                with light environment)            -   The exact eye position inclusively its probability                results from a minimum search in the previously                calculated probability mountains    -   “Parallel pupil analyzer 812”    -   Function        -   Detects based on a previously determined eye position, the            position of the pupil midpoints within the detected eyes            (thereby, the accuracy increases of the eye position, which            is important for the measurements or the subsequent            evaluation of the pupil)    -   Input        -   Initial image in original resolution as well as the            determined eye positions and face size (from the “parallel            eye analyzer” or the “parallel face finder”)    -   Output        -   Position of the pupil within the evaluated image as well as            a status indicating if a pupil was found or not    -   Detailed description        -   Based on the determined eye positions and the face size, an            image section to be processed is identified around the eye        -   Beyond this image matrix, a vector is built up containing            the minima of the image columns as well as a vector            containing the minima of the image lines        -   Within these vectors (from minimum grey values), the pupil            midpoint is as described in the following separately            detected in horizontal and vertical direction:            -   Detection of the minimum of the respective vector (as                position within the pupil)            -   Based on this minimum, within the vector, in positive                and negative direction, the position is determined, at                which an adjustable threshold related proportionally to                the dynamic range of all vector elements is exceeded            -   The midpoints of these ranges in both vectors together                form the midpoint of the pupil in the analyzed image    -   “Temporal smart smoothing filter 814”    -   Function        -   Adaptive temporal smoothing of a data series (e.g. of a            determined face coordinate), whereby dropouts, absurd values            or extreme outliers do NOT lead to fluctuations in the            smoothened data    -   Input        -   To every activation time of the module respectively one            value of the data series and the associated quality criteria            (regarding face tracking: face score and down-scaling stage,            in which the face was found)    -   Output        -   Smoothened data value (e.g. face coordinate)    -   Detailed description        -   Via a set of filter parameters, during initializing of the            filter, its behavior can be determined        -   The current input value is used for the smoothing, if it            does not fall within one of the following categories:            -   According to the associated score, it is a dropout of                the data series            -   According to the associated down-scaling stage, it is an                absurd value (value, which had been determined in                down-scaling stage which was too far away)            -   According to the too large difference towards the last                value used for the smoothing, it is an outlier        -   If one of these criteria is fulfilled, further, the            previously determined smoothened value is outputted,            otherwise, the current value is consulted for the smoothing        -   In order to obtain a possibly low delay during smoothing,            the current values are stronger rated than pas ones:            -   Currently smoothened value=current value*smoothing                coefficient+last smoothened value*(1−smoothing                coefficient)            -   The smoothing coefficient is in defined borders                dynamically adjusted to the tendency of the data to be                smoothened:                -   Reduction with rather constant value development of                    the data series                -   Increase with increasing or decreasing value                    development of the data series        -   If in the long term a larger leap regarding the ellipsis            parameters to be smoothened occurs, the filter and, thus,            also the smoothened value development adjusts    -   “3D camera system model 804 a”    -   Function        -   Modeling of the 3D room in which several cameras, the user            (or his/her eyes) and possibly a screen are located    -   Input        -   Configuration file, which contains the model parameters            (position parameters, optical parameters, et al) of all            elements of the model    -   Output        -   Provides a statistical framework and functions for the            calculations within this model    -   Detailed description        -   Modeling of the spatial position (position and rotation            angle) of all elements of the model as well as their            geometric (e.g. pixel size, sensor size, resolution) and            optical (e.g. focal length, objective distortion)            characteristics        -   The model comprises at this point in time the following            elements:            -   Camera units consisting of:                -   Camera sensors                -   Objective lenses            -   eyes            -   Display        -   Besides the characteristics of all elements of the model, in            particular the subsequently described functions “3D position            calculation” (for the calculation of the eye position) and            “3D gaze direction calculation” (for the calculation of the            gaze direction) are provided        -   In other application cases, also the following functions are            provided:            -   By means of this model, inter alia the 3D line of sight                (consisting of the pupil midpoint and the gaze direction                vector (corresponding to biology and physiology of the                human eye can be calculated            -   Optionally, also the point of view of a viewer on                another object of the 3D model (e.g. on a display) may                be calculated as well as the focused area of the viewer    -   “3D position calculation 804”    -   Function        -   Calculation of the spatial position (3D coordinates) of a            point, captured by two or more cameras (e.g. pupil midpoint)    -   Input        -   2D-coordinates of a point in two camera images    -   Output        -   3D-coordinates of the point        -   Error measure: describes the accuracy of the transferred 2D            coordinates in connection with the model parameters    -   Detailed description        -   From the transferred 2D-coordinates, by means of the “3D            camera system model” (in particular under consideration of            the optical parameters) for both cameras, the light beams            are calculated, which have displayed the 3D point as 2D            point on the sensors        -   These light beams are described as straight lines in the 3D            room of the mode        -   The point of which both straight lines have the smallest            distance (in the ideal case, the intersection of the            straight lines), is assumed to be the searched 3D point

Advantages

Determination of the face position and the eye position in a (live)camera image in 2D and by recalculation in the 3D room in 3D (byincluding of a 3D room model)

The algorithms presented under 3. are optimized to real-time capable andparallel processing in FPGAs High frame rates (60 FPS @ 640×480 on aXILINX Spartan 3A DSP @ 48 MHz) and short latency periods due toentirely parallel processing without recursion in the processingchain→very fast image processing and an output of the results with aminimum delay

-   -   Minimum construction room as the entire functionality can be        achieved with one component (FPGA)

Low energy consumption

-   -   Fixed time regime (constant time difference between consecutive        results) and thereby, predestined for the use in        security-critical applications    -   Possibility to direct porting of the processing from the FPGA to        an ASIC (application specific integrated circuit)→very cost        efficient solution at high quantities due to exploitation of the        scaling effects

Workarounds

-   -   use of other algorithms for the overall functionality of        individual sub-functions

Possibility of proof for patent infringement

-   -   Proof of patent infringement by copying/cloning of the FPGA bit        file/network lists        -   Would be easily to prove, e.g. by check amounts        -   Further, bit files can be bound to a FPGA-ID→the copying            would then only be possible by using of FPGA similar IDs    -   Proof of patent infringement by “disassembling” of the FPGA bit        file/network lists        -   Indications for patent infringements would possibly be            recognizable by disassembling of the respective FPGA bit            file/network list        -   The concrete proof could only hardly be rendered

Application

-   -   Advantages during the application compared to a software        solution        -   Autonomous functionality (System on Chip)        -   Possibility of the easy transfer into an ASIC        -   Space-saving integration into existing systems/switches    -   Application fields similar to those of a software solution (in a        (live) camera image data stream face positions and the        corresponding eye positions are detected, which are used for the        below listed applications)        -   Security applications            -   E.g. momentary nodding off warning systems in the                automotive field, by evaluation of the eyes (blink                degree) and the eyes and head movement        -   Man-machine-communication            -   E.g. input interfaces for technical devices (head or eye                position as input parameter)        -   Gaze-tracking            -   E.g. face and eye positions as preliminary stage for the                gaze direction determination (in combination with                “integrated eye-tracker”)        -   Marketing            -   E.g. assessing attractiveness of advertisement by                determining the head and eye parameters (inter alia                position)

In the following, by means of two illustrations, further backgroundknowledge regarding the above described aspects is disclosed.

A detailed calculation example for this gaze direction calculation isdescribed in the following by means of FIGS. 8a to 8 e.

Calculating the Pupil Midpoint

As already described, with depicting the circular pupil 806 a by thecamera lenses 808 a and 808 b on the image sensors 802 a and 802 b anelliptic pupil projection respectively arises (cf. FIG. 8a ). The centerof the pupil is on both sensors 802 a and 802 b and, thus, also in therespective camera images depicted as midpoint E_(MP) ^(K1) and E_(MP)^(K2) of the ellipsis. Therefore, due to stereoscopic rear projection ofthese two ellipsis midpoints E_(MP) ^(K1) and E_(MP) ^(K2), the 3D pupilmidpoint can be determined by means of the objective lens model. Anoptional requirement thereto is an ideally time synchronous picture sothat the depicted scenes taken from both cameras are identical and,thus, the pupil midpoint was collected at the same position.

Initially, for each camera, the rear projection beam RS of the ellipsismidpoint has to be calculated, which runs along an intersection beambetween the object and the intersection on the object's side (H1) of theoptical system (FIG. 8a ).

RS(t)=RS ₀ +t·RS _({right arrow over (n)})  (A1)

This rear projection beam is defined by equation (A1). It consists of astarting point RS₀ and a standardized direction vectorRS_({right arrow over (n)}), which result in the used objective lensmodel (FIG. 8b ) from the equations (A2) and (A3) from the two mainpoints H₁ and H₂ of the objective as well as from the ellipsis centerE_(MP) in the sensor plane. For this, all three points (H₁, H₂ andE_(MP)) have to be available in the eye-tracker coordination system.

$\begin{matrix}{{RS}_{0} = H_{1}} & \left( {A\; 2} \right) \\{{RS}_{ii}\frac{H_{2} - E_{MP}}{{H_{2} - E_{MP}}}} & \left( {A\; 3} \right)\end{matrix}$

The main points can be calculated by the equations

H ₂ =K _(O) b·K ₁,

and

H ₁ =K _(O)+(b+d)·K _({right arrow over (n)})

Directly form the objective lens and camera parameters (FIG. 8b ),wherein K_(O) is the midpoint of the camera sensor plane andK_({right arrow over (n)}) is the normal vector of the camera sensorplane. The 3D ellipsis center in the camera coordination system can becalculated from the previously determined ellipsis center parametersx_(m) and y_(m), which are provided by means of the equation

$P_{Camera} = {\begin{bmatrix}P_{Camera}^{x} \\P_{Camera}^{y} \\P_{Camera}^{z}\end{bmatrix} = {\left( {\begin{bmatrix}P_{image}^{x} \\P_{image}^{y} \\0\end{bmatrix} + \begin{bmatrix}S_{offset}^{x} \\S_{offset}^{y} \\0\end{bmatrix} - {\frac{1}{2} \cdot \begin{bmatrix}S_{res}^{x} \\S_{res}^{y} \\0\end{bmatrix}}} \right) \cdot S_{PxGr}}}$

Thereby, P_(image) is the resolution of the camera image in pixels,S_(offset) is the position on the sensor, at which it is started to readout the image, S_(res) is the resolution of the sensor and S_(PxGr) isthe pixel size of the sensor.

The searched pupil midpoint is in the ideal case the point ofintersection of the two rea projection beams RS^(K1) und RS^(K2). Withpractically determined model parameters and ellipsis midpoints, however,already by minimum measurement errors, no intersection point of thestraight lines result anymore in the 3D room. Two straight lines in thisconstellation, which neither intersect, nor run parallel, are designatedin the geometry as skew lines. In case of the rear projection, it can beassumed that the two skew lines respectively pass the pupil midpointvery closely. Thereby, the pupil midpoint lies at the position of theirsmallest distance to each other on half of the line between the twostraight lines.

The shortest distance between two skew lines is indicated by aconnecting line, which is perpendicular to the two straight lines. Thedirection vector {right arrow over (n)}_(St) of the perpendicularlystanding line on both rear projection beams can be calculated accordingto equation (A4) as intersection product of its direction vectors.

{right arrow over (n)} _(St) =RS _({right arrow over (n)}) ^(K1) ×RS_(n) ^(K2)  (A4)

The position of the shortest connecting line between the rear projectionbeams is defined by equation (A5). By use of RS^(K1)(s), RS^(K2)(t) and{right arrow over (n)}_(St) it results therefrom an equation system,from which s, t and u can be calculated.

RS ^(K1)(s)+u {right arrow over (n)} _(St) RS ^(K2)(t)  (A5)

The searched pupil midpoint P_(MP), which lies half the line between therear projection beams, results consequently from equation (A6) afterusing the values calculated for s and u.

$\begin{matrix}{P_{MP} = {{{RS}^{K\; 1}(s)} + {\frac{u}{2} \cdot {\overset{\rightarrow}{n}}_{St}}}} & \left( {A\; 6} \right)\end{matrix}$

As indicator for the precision of the calculated pupil midpoint,additionally a minimum distance d_(RS) between the rear projection beamscan be calculated. The more precise the model parameters and theellipsis parameters were, the smaller is d_(RS).

d _(RS) =u·|n _(St)|  (A7)

The calculated pupil midpoint is one of the two parameters, whichdetermine the line of sight of the eye to be determined by theeye-tracker. Moreover, it is needed for the calculation of the gazedirection vector P_({right arrow over (n)}), which is described in thefollowing.

The advantage of this method for calculating the pupil midpoint is thatthe distances of the cameras to the eye do not have to be firmly storedin the system. This is useful e.g. in the method described in the patentspecification of DE 10 2004 046 617 A1.

Calculation of the Gaze Direction Vector

The gaze direction vector P_({right arrow over (n)}) to be determinedcorresponds to the normal vector of the circular pupil surface and,thus, is due to the alignment of the pupil specified in the 3D room.From the ellipsis parameter, which can be determined for each of the twoellipsis-shaped projections of the pupil on the camera sensors, theposition and alignment of the pupil can be determined. Thereby, thelengths of the two half-axes as well as the rotation angles of theprojected ellipses are characteristic for the alignment of thee pupiland/or the gaze direction relative to the camera position.

One approach for calculating the gaze direction from the ellipsisparameters and firmly in the eye-tracking system stored distancesbetween the cameras and the eye is e.g. described in the patentspecification of DE 10 2004 046 617 A1. As shown in FIG. 8e , thisapproach assumes a parallel projection, whereby the straight linedefined by the sensor normal and the midpoint of the pupil projected tothe sensor passes through the pupil midpoint. For this, the distances ofthe cameras to the eye need to be previously known and firmly stored inthe eye-tracking system.

With the model of the camera objective presented in this approach, whichdescribes the display behavior of a real object, however, a perspectiveprojection of the object to the image sensor occurs. Due to this, thecalculation of the pupil midpoint can be performed and the distances ofthe camera to the eye have not to be previously known, which constitutesone of the essential improvements compared to the above mentioned patentspecification. Due to the perspective projection, however, the form ofthe pupil ellipsis displayed on the sensor results contrary to theparallel projection not only due to the inclination of the pupilvis-à-vis the sensor surface. The deflection δ of the pupil midpointfrom the optical axis of the camera objective lens likewise has, asdepicted in FIG. 8b , an influence to the form of the pupil projectionand, thus, to the ellipsis parameters determined therefrom.

Contrary to the sketch in FIG. 8b , the distance between pupil andcamera with several hundred millimeters is very large vis-à-vis thepupil radius, which is between 2 mm and 8 mm. Therefore, the deviationof the pupil projection from an ideal ellipsis form, which occurs withthe inclination of the pupil vis-à-vis the optical axis, is very smalland can be omitted.

In order to be able to calculate the gaze direction vectorP_({right arrow over (n)}), the influence of the angle g to the ellipsisparameter has to be eliminated so that the form of the pupil projectionalone is influenced by the alignment of the pupil. This is given, if thepupil midpoint P_(MP) directly lies in the optical axis of the camerasystem. Therefore, the influence of the angle δ can be removed bycalculating the pupil projection on the sensor of a virtual camerasystem νK, the optical axis of which passes directly the previouslycalculated pupil midpoint P_(MP), as shown in FIG. 8 c.

The position and alignment of such a virtual camera system 804 a′ (νK inFIG. 8c ) can be calculated from the parameter of the original camerasystem 804 a (K in FIG. 8b ) by rotation about its object-side mainpoint H₁. Thus, this corresponds simultaneously to the object-side mainpoint νH₁ of the virtual camera system 804 a′. Therefore, the directionvectors of the intersection beams of the depicted objects are in frontand behind the virtual optical system 808 c′ identically to those in theoriginal camera system. All further calculations to determining the gazedirection vector take place in the eye-tracker coordination system.

The standardized normal vector νK_({right arrow over (n)}) of thevirtual camera νK is obtained as follows:

$\begin{matrix}{{vK}_{\overset{\rightarrow}{n}} = \frac{P_{MP} - H_{1}}{{P_{MP} - H_{1}}}} & ({A8})\end{matrix}$

For the further procedure, it is useful to calculate the rotation anglesabout the x-axis (νK_(θ)), about the y-axis (νK_(φ)) and about thez-axis (νK_(ψ)) of the eye-tracker coordination system, about which theunit vector of the z-direction of the eye-tracker coordination systemabout several axes of the eye-tracker coordination system has to berotated, in order to obtain the vector νK_({right arrow over (n)}). Dueto rotation of the unit vector of the x-direction, as well as of theunit vector of the y-direction of the eye-tracker coordination systemabout the angles νK_(θ), νK_(φ) and νK_(ψ), the vectors νK_(x) andνK_({right arrow over (y)}) can be calculated, which indicate the x- andy-axis of the virtual sensor in the eye-tracker coordination system.

In order to obtain the position of the virtual camera system 804 a′(FIG. 8c ), its location vector and/or coordinate origin νK₀, which issimultaneously the midpoint of the image sensor, has to be calculated bymeans of equation (A9) in a way that it lies in the intersection beam ofthe pupil midpoint P_(MP).

νK ₀ =νH ₁−(d+b)·νK _({right arrow over (n)})  (A9)

The distance d that may be used for this purpose between the main pointsas well as the distance b between the main plane 2 and the sensor planehave to be known or e.g. determined by an experimental setup.

Further, the position of the image-side main point results from equation(A10).

νH ₂ =νH ₁ −d·νK _({right arrow over (n)})  (A10)

For calculating the pupil projection on the virtual sensor 804 a′,initially the edge points RP^(3D) of the previously determined ellipsison the Sensor in the original position may be used. These result fromthe edge points RP^(2D) of the ellipsis E in the camera image, wherebycorresponding to FIG. 8d , E_(a) is the short half-axis of the ellipsis,E_(b) is the long half-axis of the ellipsis E_(x) _(m) , and E_(y) _(m)is the midpoint coordinate of the ellipsis, and E_(α) is the rotationangle of the ellipsis. The position of one point RP^(3D) in theeye-tracker coordination system can be calculated by the equations (A11)to (A14) from the parameters of the E, the sensors S and the camera K,wherein ω indicates the position of an edge point RP^(2D) according toFIG. 8d on the ellipsis circumference.

$\begin{matrix}{\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix} = \begin{bmatrix}{E_{a} \cdot {\cos (\omega)}} \\{E_{b} \cdot {\sin (\omega)}}\end{bmatrix}} & \left( {A\; 11} \right) \\{{RP}^{2\; D} = \begin{bmatrix}{{x^{\prime} \cdot {\cos \left( E_{\alpha} \right)}} + {y^{\prime} \cdot {\sin \left( E_{\alpha} \right)}} + E_{x_{m}}} \\{{{- x^{\prime}} \cdot {\sin \left( E_{\alpha} \right)}} + {y^{\prime} \cdot {\cos \left( E_{\alpha} \right)}} + E_{y_{m}}}\end{bmatrix}} & ({A12}) \\{\begin{bmatrix}s_{1} \\t_{1}\end{bmatrix} = {\left( {{{RP}^{2\; D} \cdot \frac{1}{2} \cdot S_{res}} - S_{offset}} \right) \cdot S_{PxGr}}} & ({A13}) \\{{RP}^{3\; D} = {K_{0} + {s_{1} \cdot K_{\overset{\rightarrow}{x}}} + {t_{1} \cdot K_{\overset{\rightarrow}{y}}}}} & ({A14})\end{matrix}$

The direction of one intersection beam KS in the original camera system,which displays a pupil edge point as ellipsis edge point RP^(3D) on thesensor, is equal to the direction of the intersection beam νKS in thevirtual camera system, which displays the same pupil edge point asellipsis edge point RP^(3D) on the virtual sensor. The intersectionbeams of the ellipsis edge points in FIG. 8b and FIG. 8c demonstratethis aspect. Thus, the two beams KS and νKS have the same directionvector, which results from equation (A15). For the location vector νKS₀of the virtual sensor-side intersection beam νKS, νKS₀=νH₂ isapplicable.

$\begin{matrix}{{v\; {KS}_{\overset{\rightarrow}{n}}} = {{KS}_{\overset{\rightarrow}{n}} = \frac{{RP}^{3\; D} - H_{2}}{{{RP}^{3\; D} - H_{2}}}}} & \left( {A15} \right)\end{matrix}$

The virtual intersection beam and the virtual sensor plane, whichcorresponds to the x-y-plane of the virtual camera νK, are equated inequation (A16), whereby by resolving s₂ und t₂, the parameter of theirintersection are obtained. By these, the ellipsis edge point in pixelcoordinates in the image of the virtual camera can be calculated byequation (A17).

$\begin{matrix}{{{v\; {KS}_{0}} + {{r_{2} \cdot v}\; {KS}_{\overset{\rightarrow}{n}}}} = {K_{0} + {s_{2} \cdot K_{\overset{\rightarrow}{x}}} + {t_{2} \cdot K_{\overset{\rightarrow}{y}}}}} & \left( {A\; 16} \right) \\{{vRP}^{2D} = {{\begin{bmatrix}s_{2} \\t_{2}\end{bmatrix} \cdot \frac{1}{S_{PxGr}}} + \left( {{\frac{1}{2}S_{res}} - S_{offset}} \right)}} & ({A17})\end{matrix}$

Subsequently, from several virtual edge points νRP^(2D) the parameter ofthe virtual ellipsis νE can be calculated by means of ellipsis fitting,e.g. with the “direct least square fitting of ellipses”, algorithmaccording to Fitzgibbon et al. For this, at least six virtual edgepoints νRP^(2D) may be used, which can be calculated by using several ωin equation (A11) with the above described path.

The form of the virtual ellipsis νE determined this way, only depends onthe alignment of the pupil. Furthermore, its midpoint is in the centerof the virtual sensor and together with the sensor normal, whichcorresponds to the camera normal νK_({right arrow over (n)})t, it formsa straight line running along the optical axis through the pupilmidpoint P_(MP). Thus, the requirements are fulfilled to subsequentlycalculate the gaze direction based on the approach presented in thepatent specification of DE 10 2004 046 617 A1. Thereby, with thisapproach, it is now also possible by using the above described virtualcamera system to determine the gaze direction, if the pupil midpointlies beyond the axis of the optical axis of the real camera system,which is almost the case in real applications.

As shown in FIG. 8e , the previously calculated virtual ellipsis νE isnow accepted in the virtual main plane 1. As the midpoint of νE lies inthe center of the virtual sensor and, thus, in the optical axis, the 3Dellipsis midpoint νE′_(MP) corresponds to the virtual main point 1.Simultaneously, it is the dropped perpendicular foot of the pupilmidpoint P_(MP) in the virtual main plane 1. In the following, only theaxial ratio and the rotation angle of the ellipsis νE is used. Theseform parameters of νE thereby can be used unchanged in respect of themain plane 1, as the alignments of the x- and y-axis of the 2D sensorplane, to which they refer to, correspond to the 3D sensor plane and,thus, also to the alignment of the main plane 1.

Every picture of the pupil 806 a in a camera image can arise by twodifferent alignments of the pupil. During evaluating the pupil form,therefore, as shown in FIG. 8e , two virtual intersections νS of the twopossible straights of view with the virtual main plane 1 arise from theresults of every camera. Corresponding to the geometric ratio in FIG. 8e, the two possible gaze directions P_({right arrow over (n)},1) andP_({right arrow over (n)},2) can be determined as follows.

The distance A between the known pupil midpoint and the ellipsismidpoint νE′_(MP) is:

A=|νH ₁ −P _(MP)|  (A18)

Therefrom, r can be determined with equation A19.

$\begin{matrix}{r = {\frac{\sqrt{a^{2} - b^{2}}}{b} \cdot A}} & ({A19})\end{matrix}$

Both direction vectors r_({right arrow over (n)},1) as well asr_({right arrow over (n)},2), which are aligned from νH₁ to νS₁ as wellas to νS₂, are analogously calculated to the equations

$M_{\phi} = \begin{bmatrix}1 & 0 & 0 \\0 & {\cos (\phi)} & {- {\sin (\phi)}} \\0 & {\sin (\phi)} & {\cos (\phi)}\end{bmatrix}$ $M_{\theta} = \begin{bmatrix}{\cos (\theta)} & 0 & {\sin (\theta)} \\0 & 1 & 0 \\{- {\sin (\theta)}} & 0 & {\cos (\theta)}\end{bmatrix}$ $M_{\psi} = \begin{bmatrix}{\cos (\psi)} & {- {\sin (\psi)}} & 0 \\{\sin (\psi)} & {\cos (\phi)} & 0 \\0 & 0 & 1\end{bmatrix}$${\overset{\rightarrow}{v}}^{\prime} = {M_{\theta} \cdot M_{\phi} \cdot M_{\psi} \cdot \overset{\rightarrow}{v}}$

from νK_(θ), νK_(φ), νK_(ψ) and νE_(α):

r _({right arrow over (n)},1) =M _(θ=νK) ₀ ·M _(φ=νK) _(φ) ·M _(ψ=νK)_(ψ) _(−90°−νE) _(α) ·[1,0,0]^(T)  (A20)

r _({right arrow over (n)},2) =M _(θ=νK) ₀ ·M _(φ=νK) _(φ) ·M _(ψ=νK)_(ψ) _(+90°−νE) _(α) ·[1,0,0]^(T)  (A21)

Subsequently, both virtual intersections vS1 as well as vS2 can bedetermined and therefrom, the possible gaze directionsP_({right arrow over (n)},1) as well as P_({right arrow over (n)},2).

$\begin{matrix}{{vS}_{1} = {{vH}_{1} + {r \cdot r_{\overset{->}{n},1}}}} & ({A22}) \\{{vS}_{2} = {{vH}_{1} + {r \cdot r_{\overset{->}{n},2}}}} & ({A23}) \\{P_{\overset{->}{n},1} = \frac{{vS}_{1} - P_{MP}}{{{vS}_{1} - P_{MP}}}} & ({A24}) \\{P_{\overset{->}{n},2} = \frac{{vS}_{2} - P_{MP}}{{{vS}_{2} - P_{MP}}}} & ({A25})\end{matrix}$

In order to determine the actual gaze direction, the possible gazedirections of the camera 1 (P_({right arrow over (n)},1) ^(K1) as wellas P_({right arrow over (n)},2) ^(K1)) and the camera 2(P_({right arrow over (n)},1) ^(K2) as well asP_({right arrow over (n)},2) ^(K2)) may be used. From these fourvectors, respectively one of each camera indicates the actual gazedirection, whereby these two standardized vectors are ideally identical.In order to identify them, for all four possible combinations, thedifferences of the respectively selected possible gaze direction vectorsare formed from a vector of one camera and from a vector of the othercamera. The combination, which has the smallest difference, contains thesearched vectors. Averaged, these result in the gaze direction vectorP_({right arrow over (n)}) which is to be determined. When averaging, anearly simultaneously captured image has to be assumed so that bothcameras collected the same pupil position as well as the same alignmentand, thus, the same gaze direction.

As a degree for the accuracy of the calculated gaze direction vector,additionally, the angle w_(diff) between the two averaged vectorsP_({right arrow over (n)}) ^(K1) and P_({right arrow over (n)}) ^(K2),which indicate the actual gaze direction, can be calculated. The smallerw_(diff) is, the more precise the model parameters and ellipsismidpoints were, which had been used for the calculations so far.

$\begin{matrix}{w_{diff} = {\arccos \left( \frac{P_{\overset{->}{n}}^{K\; 1} \circ P_{\overset{->}{n}}^{K\; 2}}{{P_{\overset{->}{n}}^{K\; 1}} \cdot {P_{\overset{->}{n}}^{K\; 2}}} \right)}} & ({A26})\end{matrix}$

The points of view θ_(BW) and φ_(BW) vis-à-vis the normal position ofthe pupil (P_({right arrow over (n)}) is parallel to the z-axis of theeye-tracker coordination system) can be calculated with the equations

$\phi_{BW} = {\arcsin \left( {- P_{\overset{->}{n}}^{y}} \right)}$ and$\theta_{BW} = \left\{ \begin{matrix}{0{^\circ}} & {if} & {\left( {z = 0} \right)\left( {x = 0} \right)} \\{90{^\circ}} & {if} & {\left( {z = 0} \right)\left( {x < 0} \right)} \\{{- 90}{^\circ}} & {if} & {\left( {z = 0} \right)\left( {x > 0} \right)} \\{{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} - {180{^\circ}}} & {if} & {\left( {z < 0} \right)\left( {x < 0} \right)} \\{{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} + {180{^\circ}}} & {if} & {\left( {z < 0} \right)\left( {x \geq 0} \right)} \\{\tan \left( \frac{P_{\overset{->}{n}}^{x}}{P_{\overset{->}{n}}^{z}} \right)} & {otherwise} & \;\end{matrix} \right.$

In case that a systematic deviation of the gaze direction from theoptical axis of the eye and/or from the pupil normal should beconsidered, the corresponding angles can be added to the determinedpoints of view θ_(BW) and φ_(BW). The new gaze direction vector then hasto be calculated by means of the equation

P _({right arrow over (n)}) ′=M _(θ=θ) _(BW) _(′) ·M _(φ=φ) _(BW) _(′)·M _(ψ=0) ·{right arrow over (z)}

from the new points of view θ_(BW)′ and φ_(BW)′ and {right arrow over(z)}=[0,0,1]^(T).

With the gaze direction vector P_({right arrow over (n)}) is (besidesthe pupil midpoint P_(MP) from equation A6), also the second parameterof the line of sight (LoS) which is to be determined by the 3D imageanalyzer, is known. This is derivable from the following equation.

LoS=P _(MP) +t·P _({right arrow over (n)}).

The implementation of the above introduced method does not depend on theplatform so that the above introduced method can be performed ondifferent hardware platforms, as e.g. a PC.

Development of a method for the processing of the feature extractionmethod

The objective of the present subsequent embodiments is to develop on thebasis of the parallel Hough transformation a robust method for thefeature extraction. For this, the Hough core is revised and a method forthe feature extraction is presented, which reduces the results of thetransformation and breaks them down to a few “feature vectors” perimage. Subsequently, the newly developed method is implemented in aMATLAB toolbox and is tested. Finally, an FPGA implementation of the newmethod is presented.

Parallel Hough transformation for straight lines and circles

The parallel Hough transformation uses Hough cores of different size,which have to be configured by means of configuration matrices for therespective application. The mathematic contexts and methods forestablishing such configuration matrices, are presented in thefollowing. The MATLAB alc_config_lines_curvatures.m refers to thesemethods and establishes configuration matrices for straight lines andhalf circles of different sizes.

For establishing the configuration matrices, it is initially useful tocalculate arrays of curves in discrete presentation and for differentHough cores. The requirements (establishing provisions) for the arraysof curves had already been demonstrated. Under consideration of theseestablishing provisions, in particular straight lines and half circlesare suitable for the configuration of the Hough cores. For the gazedirection determination, Hough cores with configurations for halfcircles (or curves) are used. For reasons of completeness, also theconfigurations for straight lines (or straight line segments) arederived here. The mathematic contexts for determining the arrays ofcurves for straight lines are demonstrated.

Starting point for the calculation of the arrays of curves for straightlines is the linear straight equation in (B1).

y=m·x+n  (B1)

The arrays of curves can be generated by variation of the increase m.For this, the straight line increase of 0° to 45° is broke down intointervals of same size. The number of intervals depends on the Houghcore size and corresponds to the number of Hough core lines. Theincrease may be tuned via the control variable Y_(core) of 0 tocore_(height).

$\begin{matrix}{m = {\frac{1}{{core}_{heigt}} \cdot y_{core}}} & ({B2})\end{matrix}$

The function values of the arrays of curves are calculated by variationof the control variable (in (B3) exchanged by x_(core)), the values ofwhich are of 0 to core width.

$\begin{matrix}{y = {\frac{1}{{core}_{heigt}} \cdot y_{core} \cdot x_{core}}} & ({B3})\end{matrix}$

For a discrete demonstration in the 2D plot, the function values have tobe rounded. The calculation of the arrays of curves for half circles isoriented on (Katzmann 2005, p. 37-38) and is shown in FIG. 9 b.

Starting point for the calculation of the arrays of curves is the circleequation in the coordinate format.

r ²=(x−x _(M))²+(y−y _(m))²  (B4)

With x_(M)=0 (position of the circle center on the y-axis), x=x_(core)and converting toy for the function values of the arrays of curvesfollows (B5).

y=√{square root over (r ² −x _(core) ² +y _(M))}  (B5)

As y_(M) and r are not known, they have to be replaced. For this, themathematic contexts in (B6) and (B7) from FIG. 9b may be derived.

$\begin{matrix}{y_{M} = {h - r}} & ({B6}) \\{r_{2} = {y_{M}^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B7})\end{matrix}$

By converting of (B7) to y_(M) and the condition that y_(M) has to benegative (cf. FIG. 9b ), (B8) is obtained.

$\begin{matrix}{y_{M} = \sqrt{r^{2} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}} & ({B8})\end{matrix}$

Using (B8) in (B5) leads to (B9).

$\begin{matrix}{y = \sqrt{r^{2} - x_{core}^{2} + \sqrt{r^{2}} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}} & ({B9})\end{matrix}$

From FIG. 9b , it becomes clear that the Hough core is hub-centered andlies in the y-axis of the circle coordinate system. The variablex_(core) normally runs from 0 to core_(width)−1 and, thus, has to becorrected by

$- {\frac{{core}_{width}}{2}.}$

$\begin{matrix}{y = \sqrt{r^{2} - x_{core}^{2} - \left( \frac{{core}_{width}}{2} \right)^{2} + {\cdot \sqrt{r^{2} - {\left( \frac{{core}_{width}}{2} \right)^{2} \cdot \left( {- 1} \right)}}}}} & ({B10})\end{matrix}$

Yet, the radius is missing, which is obtained by using of (B6) in (B7)and by further conversions.

$\begin{matrix}{r^{2} = {\left( {h - r} \right)^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B11}) \\{r^{2} = {h^{2} - {2{hr}} + r^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}} & ({B12}) \\{r = \frac{{h\; 2} + \left( \frac{{core}_{width}}{2} \right)^{2}}{2 \cdot h}} & ({B13})\end{matrix}$

For producing the arrays of curves, finally, the variable h of 0 to

$\frac{{core}_{height}}{2}$

has to be varied. This happens via the control variable y_(core) whichruns from 0 to core_(height).

$\begin{matrix}{r = \frac{\left( \frac{y_{core}}{2} \right)^{2} + \left( \frac{{core}_{width}}{2} \right)^{2}}{2 \cdot \frac{y_{core}}{2}}} & ({B14})\end{matrix}$

As already regarding the straight lines, the y-values for a discretedemonstration have to be rounded in the 2D plot. The arrays of curvesfor Hough core of type 2 can easily be determined by the equation (B15).

y _(Typ) _(_) ₂=core_(height) −y _(Typ) _(_) ₁  (B15)

Based on the arrays of curves, for all Hough sizes respectively twoconfigurations (type 1 and type 2) for straight lines and circles can bedetermined. The configurations are thereby determined directly from thearrays of curves (cf. Katzmann 2005, p. 35-36). Configuration matricesmay be occupied either by zeros or ones. A one thereby represents a useddelay element in the Hough core. Initially, the configuration matrix isinitialized in the dimensions of the Hough core with zero values.Thereafter, the following steps are passed:

-   1. Start with the first curve of the arrays of curves and test the    y-value of the first x-index number. If the y-value is greater zero,    then occupy in the same line (same y-index) at exactly the same    position (same x-index) the element of the configuration matrix with    one.-   2. Modify the y-values with same x-index via all curves of the array    of curves. If in the first step an element was occupied with one,    then subtract one of all y-values. If in the first step the element    was not occupied, then do nothing.-   3. Pass through steps 1 and 2 as long as all elements of the    configuration matrix were approached.

In FIG. 9c , the configuration procedure is gradually demonstrated.

Finally, I would like to respond to some peculiarities of the Hough coreconfiguration. The configurations for straight lines represent onlystraight line segments depending on the width of the Hough cores. Longerstraight line segments in the binary edge image have optionally beassembled from several detected straight line segments. The resolutionof the angles (or increase) of the straight line segments depends on theheight of the Hough core.

The configurations for circles represent circle arcs around the vertexof the half circle. Only the highest y-index number of the arrays ofcurves (smallest radius) represents a complete half circle. Thedeveloped configurations can be used for the new Hough core.

Revision of the Hough Cores

A decisive disadvantage of the FPGA implementation of Holland-Nell isthe rigid configuration of the Hough cores. The delay lines have to beparameterized prior to the synthesis and are afterwards fixedlydeposited in the hardware structures (Holland-Nell, p. 48-49). Changesduring runtime (e.g. Hough core size) are not possible any more. The newmethod is to become more flexible at this point. The new Hough coreshall be—also during runtime—in the FPGA completely newly configurable.This has several advantages. On the one hand, not two Hough cores (type1 and type 2) have to be parallel filed and on the other hand, alsodifferent configuration for straight lines and half circles may be used.Furthermore, the Hough core size can be flexibly changed during runtime.

Previous Hough core structures consist of a delay and a bypass and priorto the FPGA synthesis, it is determined, which path is to be used. Inthe following, this structure is extended by a multiplexer, a furtherregister for the configuration of the delay elements (switching themultiplexers) and by a pipeline delay. The configuration register may bemodified during runtime. This way, different configuration matrices canbe brought into the Hough core. By setting the pipeline delays, thesynthesis tool in the FPGA has more liberties during the implementationof the Hough core design and higher clock rates can be achieved.Pipeline delays break through time-critical paths within the FPGAstructures. In FIG. 9d , the new design of the delay elements aredemonstrated.

In comparison to the previous implementation according to Katzmann andHolland-Nell, the delay elements of the new Hough cores are built up abit more complex. For the flexible configuration of the delay element,an additional register may be used and the multiplexer occupies furtherlogic resources (may be implemented in the FPGA in an LUT). The pipelinedelay is optional. Besides the revision of the delay elements, alsomodifications of the design of the Hough core had been carried out. Thenew Hough core is demonstrated in FIG. 9 e.

In contrast to the previous Hough core, initially a new notation is tobe implemented. Due to an about 90° rotated design in FIG. 9e , the“line amounts”, originally referred to as signals of the initialhistogram, are as of now referred to as “column amounts”. Every columnof the Hough cores, thus, represents a curve of the arrays of curves.The new Hough core furthermore can be impinged with new configurationmatrices during runtime. The configuration matrices are filed in theFPGA-internal BRAM and are loaded by a configuration logic. This loadsthe configurations as column-by-column bit string in the chainedconfiguration register (cf. FIG. 9d ). The reconfiguration of the Houghcores involves a certain time period and depends on the length of thecolumns (or the amount of delay lines). Thereby, every column elementinvolves a clock cycle and a latency of few tack cycles by the BRAM andthe configuration logic is added. Although, the overall latency for thereconfiguration is disadvantageous, but for the video-based imageprocessing, it can be accepted. Normally, the video data streamsrecorded with a CMOS sensor have a horizontal and a vertical blanking.The reconfiguration, thus, can occur without problems in the horizontalblanking time. The size of the Hough core structure implemented in theFPGA, also pre-determines the maximally possible size of the Hough coreconfiguration. If small configurations are used, these are alignedvertically centered and in horizontal direction at column 1 of the Houghcore structure (cf. FIG. 9f ). Not used elements of the Hough corestructure, are all occupied with delays. The correct alignment ofsmaller configurations is important for the correction of thex-coordinates (cf. formulas (B17) to (B19)).

The Hough core is as previously fed with a binary edge image passingthrough the configured delay lines. With each processing step, thecolumn amounts are calculated via the entire Hough core and arerespectively compared with the amount signal of the previous column. Ifa column provides a higher total value, the total value of the originalcolumn is overwritten. As initial signal, the new Hough core provides acolumn total value and the associated column number. On the basis ofthese values, later on, a statement on which structure was found(represented by the column number) and with which appearance probabilitythis was detected (represented by the total value) can be made. Theinitial signal of the Hough cores can also be referred to as Hough roomor accumulator room. In contrast to the usual Hough transformation, theHough room is available to the parallel Hough transformation in theimage coordinate system. This means that for every image coordinate, atotal value with associated column number is outputted. For the completetransformation of the eye image, respectively one Hough core of type 1and type 2 of the non-rotated and the rotated image has to be passedthrough. Therefore, after the transformation, not only column amountwith associated column number, but also the Hough core type and thealignment of the initial image (non-rotated or rotated) are available.Furthermore, different Hough core sizes and configurations may be usedrespectively for the straight lines and half circles. Thereby, besidesthe mentioned results, also the curve type and the Hough core size canbe indicated. In summary, a result data set of the new Hough core sizeis illustrated in the following table. Regarding the parallel Houghtransformation, for every image point such a data set arises.

Description x-coordinate Is delayed according to the length of the Houghcore structure. A precise correction of the x-coordinate can take place.y-coordinate Is corrected according to the height of the Hough corestructure with$y_{new} = {y_{old} + {\left( \frac{{Number}\mspace{14mu} {of}\mspace{14mu} {lines}}{2} \right).}}$With an even number of lines, it cannot be a exactly determined middleline. With an uneven number of lines, it has be rounded up, in order toobtain the center line. column amount Appearance probability for thesearched structure (maximum value = size of the column, high valuesrepresent a high appearance probability) column number To the totalvalue associated column number (repre- sents the curve of the halfcircle or the increase of the straight line) Hough core type 0 if type 1Hough core configuration and 1 if type 2 Hough core-configuration Imagerotation 0 if initial image does not rotate and 1 if the initial imagerotates Hough core size Size of the Hough core, which has been used forthe transformation Curve type 0 if straight line configuration and 1 ifhalf circle configuration n

Overview of the result data set arising for every point of view of theinitial image with the parallel Hough transformation with revised Houghcore structure.

In contrast to the binary and threshold-based output of the Hough coresof Katzmann and Holland-Nell, the new Hough core structure producessignificantly more initial data. As such a data quantity is only hard tobe handled, a method for feature extraction is presented, which clearlyreduces the result data quantity.

Type 2 Hough Core and Image Rotation

To the embodiments regarding the parallel Hough transformation, thenecessity of the image rotation and the peculiarities of type 2 Houghcores, was already introduced. Regarding the parallel Houghtransformation, the initial image has to pass the Hough core four times.This may be useful so that the straight lines and half circles can bedetected in different angle positions. If only a type 1 Hough core isused, the image would have to be processed in the initial position androtated about 90°, 180°, and 270°. By including the type 2 Hough core,the rotation about 180° and 270° are omitted. If the non-rotated initialimage is processed with a type 2 Hough core, this corresponds to aprocessing of the about 180° rotated initial image with a type 1 Houghcore. It is similar with the rotation about 270°. This can be replacedby the processing of the about 90° rotated image with a type 2 Houghcore. For an FPGA implementation, the omission of additional rotationshas a positive effect, as image rotations normally are only solved bymeans of an external storage. According to the applied hardware, only acertain band width (maximally possible data rate) is available betweenFPGA and storage component. Regarding the use of a type 2 Hough core,the band width of the external storage component is only occupied with arotation of about 90°. Regarding the previous implementation ofHolland-Nell, it was useful to file a Hough core of type 1 and a Houghcore of type 2 in the FPGA. With the revised Hough core design, it isnow also possible to file the Hough core structure once in the FPGA andto upload configurations of type 1 or type 2. Due to this newfunctionality, the initial image can be completely transformed with onlyone Hough core and with only one image rotation.

It is still to be considered that during the processing with only oneHough core, also the quadruplicate data rate occurs in the Hough core.Regarding a video data stream of 60 fps and VGA resolution, the pixeldata rate amounts to 24 Mhz. In this case, the Hough core would have tobe operated with 96 Mhz, which already constitutes a high clock rate foran FPGA of the Spartan 3 generation. In order to optimize the design, itshould be intensified operated with pipeline delays within the Houghcore structure.

Feature Extraction

The feature extraction works on behalf of the data sets from theprevious table. These data sets can be summarized in a feature vector(B16). The feature vector can in the following be referred to as Houghfeature.

MV[MV _(X) ,MV _(Y) ,MV ₀ ,MV _(KS) ,MV _(H) ,MV _(G-1) ,MV _(A)]  (B16)

A feature vector respectively consists of respectively an x- andy-coordinate for the detected feature (MV_(x) und MV_(y)), theorientation MV₀, the curve intensity MV_(KS), the frequency MV_(H), theHough core size MV_(G-1) and the kind of the detected structure MV_(A).The detailed meaning and the value range of the single elements of thefeature vector can be derived from the following table.

MVx and MVy Both coordinates respectively run to the size of the initialimage MV₀ The orientation represents the alignment of the Hough core.This is composed by the image rotation and the used Hough core type andcan be divided into four sections. The conversion of the four sectionsinto their respective orientation is demonstrated in the followingtable. MV_(KS) The curve intensity maximally runs to the size of theHough core and corresponds to the Hough core column with the highestcolumn amount (or frequency MV_(H)). By way of illustration, it isreferred to FIG. 9e in combination with the above table. Regardingstraight lines configuration of the Hough cores, the Hough core columnrepresents the increase or the angle of the straight lines. If halfcircle configurations are used, the Hough core column represents theradius of the half circle. MV_(H) The frequency is a measure for thecorrelation of the image content with the searched structure. Itcorresponds to the column amount (cg. FIG. 9e and above table) and canmaximally reach the size of the Hough core (more precisely the size of aHough core column with non-square Hough cores). MV_(G-1) Size of theHough core used for the transformation minus one. MV_(A) Represents thekind of the detected structure according to the used Hough coreconfiguration (configuration for the straight lines = 0 or configurationfor circles = 1).

Elements of the Hough feature vector, their meaning and value range.

Straight lines Circles Orientation Orientation MV₀ Range Angle area MV₀Range Angle 0 Range 1r  0°-45° 0 Range 2 0° 1 Range 2 45°-90° 1 Range 1r90° 2 Range 1  90°-135° 2 Range 1 180° 3 Range 2r 135°-180° 3 Range 2r270°

Calculation of the orientation depending on the image rotation and theHough core type used for the transformation.

From the above tables, it becomes obvious that both elements MV₀ andMV_(KS) regarding straight lines and half circles have differentmeanings. Regarding straight lines, the combination from orientation andcurve intensity forms the position angle of the detected straight linesegment in the angle of 0° to 180°. Thereby, the orientation addressesan angle area and the curve intensity represents the concrete anglewithin this range. The greater the Hough core (more precise, the moreHough core columns are available), the finer the angle resolution is.Regarding half circles, the orientation represents the position angle orthe alignment of the half circle. Half circles may as a matter ofprinciple only be detected in four alignments. Regarding half circleconfigurations, the curve intensity represents the radius.

Besides the orientation MV₀ and the curve intensity MV_(KS), a furtherspecial feature is to be considered regarding the coordinates (MV_(x)und MV_(y)) (cf. FIG. 9g ). Regarding straight lines, the coordinatesare to represent the midpoint and regarding half circles or curves, thevertex. With this presupposition, the y-coordinate may be correctedcorresponding to the implemented Hough core structure and does notdepend on the size of the configuration used for the transformation (cf.FIG. 9f ). Similar to a local filter, the y-coordinate is indicatedvertically centered. For the x-coordinate, a context via the Hough corecolumn is established, which has provided the hit (in the featurevector, the Hough core column is stored with the designation MV_(KS)).Dependent on the Hough core type and the image rotation, alsocalculation provisions for three different cases can be indicated. For aHough core of type 1, it is respectively referred to formula (B17) forthe non-rotated and the rotated initial image. If a Hough core of type 2is available, it has to be referred to formula (B18) or formula (B19)dependent on the image rotation.

$\begin{matrix}{\mspace{79mu} {{MV}_{x_{corrected}} = {{MV}_{x_{detected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}}}} & ({B17}) \\{{MV}_{x_{corrected}} = {{imagewidth}_{{non} - {rotated}} - \left( {{MV}_{x_{detected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}} \right)}} & ({B18}) \\{{MV}_{x_{corrected}} = {{imagewidth}_{rotated} - \left( {{MV}_{x_{detected}} + {{floor}\left( \frac{\left( {MV}_{{KS} + 1} \right)}{2} \right)}} \right)}} & ({B19})\end{matrix}$

With the instruction “floor”, the fractional rational number is roundedoff. In the FPGA, this corresponds to the simple cutting of binarydecimals. After the orientation had been determined and the coordinatesof the Hough features had been corrected, the actual feature extractioncan take place.

For the feature extraction, three threshold values in combination with anon-maximum suppression operator are used. The non-maximum suppressionoperator differs regarding straight lines and half circles. Via thethreshold values, a minimum MV_(KS) _(min) and maximum curve intensityMV_(KS) _(max) is given and a minimum frequency MV_(H) _(min) isdetermined. The non-maximum suppression operator can be seen as being alocal operator of the size 3×3 (cf. FIG. 9h ). A valid feature for halfcircles (or curves) arises exactly if the condition of the non-maximumsuppression operator (nms-operator) in (B23) is fulfilled and thethresholds according to formulas (B20) to (B22) are exceeded.

MV _(nms) _(2,2) ^(KS) ≧MV _(KS) _(min)   (B20)

MV _(nms) _(2,2) ^(KS) ≧MV _(KS) _(max)   (B21)

MV _(nms) _(2,2) ^(H) ≧MV _(H) _(min)   (B22)

MV _(nms) _(1,1) ^(H)

MV _(nms) _(1,2) ^(H)

MV _(nms) ₃ ^(H) MV _(nms) _(2,1) ^(H) MV _(nms) _(2,3) ^(H) MV _(nms)_(3,1) ^(H) MV _(nms) _(3,2) ^(H) MV _(nms) _(3,3) ^(H) MV _(nms) _(2,2)^(H)  (B23)

Due to the non-maximum suppression, Hough features are suppressed, whichdo not constitute local maxima in the frequency room of the featurevector. This way, Hough features are suppressed, which do not contributeto the searched structure and which are irrelevant for thepost-processing. The feature extraction is only parameterized via threethresholds, which can be beforehand usefully adjusted. A detailedexplanation of the thresholds can be derived from the following table.

Comparable parameter of the method Threshold according value Descriptionto Katzmann MV_(H) _(min) Threshold value for a minimum Hough-Thresfrequency, i.e. column total value, which is not allowed to fall below.MV_(KS) _(min) Threshold value for a minimum Bottom-Line curve of theHough feature. With Hough cores with straight line configuration, thethreshold relates to the angle area detected by the Hough core. MV_(KS)_(max) Behaves like MV_(KS) _(min) but Top-Line for a maximum.

Detailed description of the three threshold values for the extraction ofHough features from the Hough room. Compared to the method according toKatzmann, the parameters are indicated with similar function.

Regarding straight lines, a non-maximum suppression operator of the size3×3 (cf. FIG. 9h ) can be likewise deduced. Thereby, some peculiaritiesare to be considered. Unlikely to the curves, the searched structuresregarding the straight line segments are not detected according tocontinuously occurring of several maxima along the binary edgedevelopment. The non-maximum suppression, thus, can be based on themethod in the Canny edge detection algorithm. According to the Houghcore type and the detected angle area, three cases can be distinguished(cf. FIG. 9i in combination with the above table). The casedistinguishing is valid for rotated as well as for non-rotated initialimages, as the retransformation of rotated coordinates only takes placeafter the non-maximum suppression. Which nms-operator is to be used,depends on the Hough core type and on the angle area, respectively. Theangle area provided by a Hough core with configuration for straightlines is divided by the angle area bisection. The angle area bisectioncan be indicated as Hough core column (decimally refracted) (MV_(KS)_(halbe) ). The mathematical context depending on the Hough core size isdescribed by formula (B24). In which angle area the Hough feature islying, refers to the Hough core column having delivered the hit(MV_(KS)), which can be directly compared to the angle area bisectionalHough core column.

$\begin{matrix}{{MV}_{{KS}_{half}} = {{\tan \left( \frac{45}{2} \right)} \cdot \frac{\pi}{180} \cdot {Houghcore}_{size}}} & ({B24})\end{matrix}$

If an operator has been selected, the condition regarding the respectivenms-operator can be requested similar to the non-maximum suppression forcurves (formulas (B25) to (B27)). If all conditions are fulfilled and ifadditionally the threshold values according to the formulas (B20) to(B22) are exceeded, the Hough feature at position nms_(2,2) can beassumed.

Hough core type Angle area nms-operator Condition Range 1a 1 A 1 MV_(KS)≦ MV_(KS) _(half) Range 1b 1 B 2 MV_(KS) > MV_(KS) _(half) Range 2a 2 A1 MV_(KS) ≦ MV_(KS) _(half) Range 2b 2 B 3 MV_(KS) > MV_(KS) _(half)

Decision on one nms-operator depending on the Hough core tye and theangle area, in which the hit occurred.

(MV _(nms) _(2,2) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(2,3) ^(H) >MV _(nms) _(2,2) ^(H))  (B25)

(MV _(nms) _(1,3) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(3,1) ^(H) >MV _(nms) _(2,2) ^(H))  (B26)

(MV _(nms) _(1,1) ^(H) >MV _(nms) _(2,2) ^(H))

(MV _(nms) _(3,3) ^(H) >MV _(nms) _(2,2) ^(H))  (B27)

The completion of the feature extraction forms the re-rotation of the x-and the y-coordinates of rotated Hough features. For thepost-processing, these should again be available in the imagecoordination system. The retransformation is regardless of the curvetype (irrelevant if straight line or curve) to be executed, if therotated initial image is processed. In the formulas (B28) and (B29), themathematical context is described. With image width, the width of thenon-rotated initial image is meant.

MV _(y) =MV _(x) _(rot)   (B28)

MV _(x)=imagewidth−MV _(y) _(rot)   (B29)

By means of the feature extraction, it is possible to reduce the resultdata of the parallel Hough transformation up to a few points. These maythen be transferred to the post-processing as feature vector.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

1. A 2D image analyzer with the following features: an image scaler,which is configured to receive an image, which comprises a searchedpattern and to scale the received image according to a scaling factor;an image generator, which is configured to produce an overview image,which comprises a plurality of copies of the received and scaled image,wherein each copy is scaled about a different scaling factor; whereinsaid producing comprises compiling the copies of the received and scaledimage to form the overview image and arranging the copies of thereceived and scaled image within the image matrix of the overview image,said image generator calculating, for the copies of the received andscaled image, a respective position within the overview image whiletaking into account a gap between the copies of the received and scaledimage within the overview image and a gap of the copies of the receivedand scaled image to one or more of the borders of the overview image;and a pattern finder, which is implemented on an FPGA with parallelarchitectures and is configured to compare a predetermined pattern withthe overview image and to output an information regarding the positionwithin the overview image, at which an accordance between the searchedpattern and the predetermined pattern is maximal, wherein the positionrelates to a respective copy of the received and scaled image, thepattern finder searching all of the scaling stages which are of interestall at once in one step only.
 2. The 2D image analyzer according toclaim 1, wherein every scaled image is assigned to a respective positionin the overview image according to the respective scaling factor.
 3. The2D image analyzer according to claim 1, wherein the respective positionis calculable by an algorithm, which considers a gap between the scaledimages in the overview image, a gap of the scaled images to one or moreborders of the overview image and/or other predefined conditions.
 4. The2D image analyzer according to claim 1, wherein the pattern findercomprises a feature transformer for the transformation of one or morefeatures and a classifier, which is configured to classify patterns inthe overview image or to classify a pattern in the overview image, whichis transmitted in a feature room by the feature transformer.
 5. The 2Dimage analyzer according to claim 4, wherein the feature transformer isconfigured to extract features, wherein the feature transformation isbased on a Hough algorithm, census algorithm, gradient algorithm,histogram-based algorithm, and/or on correlation-based algorithms. 6.The 2D image analyzer according to claim 4, wherein the classifier isconfigured to carry out the classification based on a census-transformedversion of the overview image or on the basis of a version of theoverview image transmitted to a feature room or on the basis of aversion of the overview image transmitted into a gradient image.
 7. The2D image analyzer according to claim 6, wherein the pattern finder isconfigured to identify one or more local maxima in thecensus-transformed version of the overview image or in the version ofthe overview image transmitted into the feature room or in the versionof the overview image transmitted into the gradient image, wherein aposition of a local maximum indicates the position of the predeterminedpattern to be identified in the respective copy of the received andscaled image.
 8. The 2D image analyzer according to claim 6, wherein thepattern finder comprises an amount filter and/or a maximum filter, whichis configured to smoothen the position of the identified predeterminedpattern in the respective copies of the received and scaled image and/orto smoothen and correct the position of the local maxima in theclassified overview image.
 9. The 2D image analyzer according to claim4, wherein the pattern finder carries out the feature transformation andclassification for each received and scaled image in order to determinethe position of the local maximum, which indicates the position of theidentified predetermined pattern in the respective copy of the receivedand scaled image, on the basis of averaging.
 10. The 2D image analyzeraccording to claim 4, wherein the position of the identifiedpredetermined pattern relates to a reference point of a face, an eye orgenerally to the searched object.
 11. The 2D image analyzer according toclaim 1, wherein the 2D image analyzer is connected to a processing unitcomprising a selective adaptive data processor, which is configured toreceive several sets of values, wherein every set is assigned to arespective sample and which comprises a filter processor, wherein thefilter processor is configured to output plausible sets on the basis ofthe received sets and, thus, in a way that an implausible set isreplaced by a plausible set.
 12. The 2D image analyzer according toclaim 1, wherein the 2D image analyzer is connected to a processing unitcomprising a 3D image analyzer, which is configured to receive at leastone first set of image data, which is determined on the basis of a firstimage and a further set of image which is determined on the basis of thefirst image or a further image, wherein the first image displays apattern of a three-dimensional object from a first perspective in afirst image plane and wherein the further set comprises a relativeinformation, which describes a relative relation between a point of thethree-dimensional object and the first image plane, wherein the 3D imageanalyzer comprises the following features: a position calculator, whichis configured to calculate a position of the pattern in athree-dimensional room based on a first set, a further set which isdetermined on the basis of the further image, or in order to calculatethe position of the pattern in the three-dimensional room based on thefirst set and a statistically evaluated relation between twocharacteristic features in the first image to each other; and analignment calculator, which is configured to calculate two possible 3Dgaze vectors and to determine from the two possible 3D gaze vectors the3D gaze vector, according to which the pattern in the three-dimensionalroom is aligned, wherein the calculation and determination are based onthe first set, the further set and on the position of the patter to becalculated.
 13. A 2D image analyzing system comprising the 2D imageanalyzer according to claim 9, wherein the 2D image analyzer isconnected to an image analyzing system for collecting and/or tracking ofa pupil comprising a first Hough path for a first camera and aprocessing unit for post-processing of the first Hough path results,wherein the first Hough path comprises a Hough processor with thefollowing features: a pre-processor, which is configured to receive aplurality of samples, each comprising an image and in order to rotateand/or reflect the image of the respective sample and in order to outputa plurality of versions of the image of the respective sample for eachsample; and a Hough transformation unit, which is configured to collecta predetermined searched pattern in the plurality of samples on thebasis of the plurality of versions, wherein a characteristic beingdependent on the searched pattern of the Hough transformation unit isadjustable, wherein the processing unit comprises a unit for analyzingthe collected pattern and for outputting a set of geometry parameters,which describes a position and/or a geometry of the pattern for eachsample.
 14. A 2D image analyzing system comprising the 2D image analyzeraccording to claim 9 and an evaluation unit, wherein the evaluation unitis configured to detect an absent reaction of the eye.
 15. A method foranalyzing a 2D image, comprising: scaling a received image comprising asearched pattern according to a scaling factor; producing an overviewimage comprising a plurality of copies of the received and scaled image,wherein every copy is scaled about a different scaling factor; whereinsaid producing comprises compiling the copies of the received and scaledimage to form the overview image and arranging the copies of thereceived and scaled image within the image matrix of the overview image,said image generator calculating, for the copies of the received andscaled image, a respective position within the overview image whiletaking into account a gap between the copies of the received and scaledimage within the overview image and a gap of the copies of the receivedand scaled image to one or more of the borders of the overview image;and comparing, by means of a pattern finder implemented on an FPGA withparallel architectures, a predetermined pattern with the overview imageand outputting an information in respect to a position within theoverview image at which an accordance between the searched pattern andthe predetermined pattern is maximal, wherein the position relates to arespective copy of the received and scaled image; the pattern findersearching all of the scaling stages which are of interest all at once inone step only.
 16. A non-transitory digital storage medium having acomputer program stored thereon to perform the method for analyzing a 2Dimage, comprising: scaling a received image comprising a searchedpattern according to a scaling factor; producing an overview imagecomprising a plurality of copies of the received and scaled image,wherein every copy is scaled about a different scaling factor; whereinsaid producing comprises compiling the copies of the received and scaledimage to form the overview image and arranging the copies of thereceived and scaled image within the image matrix of the overview image,said image generator calculating, for the copies of the received andscaled image, a respective position within the overview image whiletaking into account a gap between the copies of the received and scaledimage within the overview image and a gap of the copies of the receivedand scaled image to one or more of the borders of the overview image;and comparing, by means of a pattern finder implemented on an FPGA withparallel architectures, a predetermined pattern with the overview imageand outputting an information in respect to a position within theoverview image at which an accordance between the searched pattern andthe predetermined pattern is maximal, wherein the position relates to arespective copy of the received and scaled image; the pattern findersearching all of the scaling stages which are of interest all at once inone step only, when said computer program is run by a computer.